Array Network Facility

As we begin our fourth year of operations, the Array Network Facility (ANF) for the EarthScope USArray Transportable Array seismic network has continued the timely delivery of metadata and waveform data from the growing number of Transportable Array stations. The network has increased in size to 307 stations with 244 out of the 400 new TA sites installed (as of February 2007). Starting next spring, equipment will start to roll from current stations to new locations to the east of the current footprint. Use of the Antelope software package has allowed us to maintain and operate such a dynamic network configuration, facilitating the collection and transfer of data, the generation and merging of the metadata as well as the real-time monitoring of state of health of TA station data-loggers and their command and control. Four regional networks (Anza, BDSN, SCSN, and UNR) as well as the USNSN contribute data to the Transportable Array in real-time. Although the real-time data flow to the IRIS DMC has been consistently above 90% in the last year, the ANF has strived to recover the remaining 10% of data to the DMC by connecting to a data storage device (Baler) at each station to generate day long seed files: automation of this process is ongoing. Operation of the USArray at the ANF has benefited by the real-time interface with the ORB and the Datascope database using PHP for display on the ANF website (http://anf.ucsd.edu). A recent feature in the website is round-robin database (rrd) plots allowing for scoping of individual and grouped state of health parameters. Information available for all stations includes: location, maps, photographs, equipment deployed, communications, distribution of events recorded by each station, and displays of daily, weekly, and monthly station noise spectra as generated by the DMC. Analyst review of automatic locations for the USArray network is being done at the ANF to monitor data quality. All events are associated with the USGS and regional network bulletins. As of February 2007, around 13,000 weekly picks are being fully reviewed by analysts at the ANF and over 19,000 events have been recorded. We find a small percentage of events that cannot be associated with existing bulletins. This information is used by the regional network operators to help them determine which stations may be beneficial to permanently add to their seismic networks.

The Transportable Array (TA) component of USArray uses real-time telemetry to send data to the Array Network Facility (ANF) through a variety of satellite, mobile phone, wireless and wired communication links. The ANF is responsible for the timely delivery of metadata and waveform data to the IRIS DMC from the growing number of Transportable Array stations. The IRIS DMC makes these data available to the research community. The network has increased in size to 327 stations with 259 out of the 400 new TA sites installed (as of 28 February 2007). Starting in Fall 2007, equipment will start to roll from current stations to new locations to the east of the current footprint. Use of the Antelope software package has allowed the ANF to maintain and operate this extremely dynamic network configuration, facilitating the collection and transfer of data, the generation and merging of the metadata as well as the real-time monitoring of state of health of TA station data-loggers and their command and control. Four regional networks (ANZA, BDSN, SCSN, and UNR) as well as the USNSN contribute data to the Transportable Array in real-time. Although the real-time data flow to the IRIS DMC has been 93.4% over the last year, the ANF and the TA field teams have extended every effort and have managed to recover an additional 4.8% by recovering data from the local data storage device (Baler 14) at each station. Once the missing data is recovered, we then generate station-channel-day volume seed files, which are resent to the DMC to bring the total data recovery rate to 98.4%. The total network uptime is above 99%. Analyst review of automatic locations for the USArray network is being done at the ANF as part of the data quality monitoring strategy. All events are associated with the USGS and regional network bulletins. As of February 2007, around 13,000 weekly picks are being fully reviewed by analysts at the ANF and over 19,000 events have been recorded. We find a small percentage (about 10 %) of events that cannot be associated with existing bulletins. This information is used by the regional network operators to help them determine which TA stations may be beneficial to permanently add to their seismic networks. Operation of the USArray at the ANF has benefited by the real-time interface with the ORB and the Datascope database using PHP for display on the ANF website (http:anf.ucsd.edu) to provide station and system state-of- health information to field teams. Information available for all stations includes: location, maps, photographs, equipment deployed, communications, distribution of events recorded by each station, and displays of daily, weekly, and yearly state of health parameters as well as station noise spectra generated by the DMC.

The Array Network Facility website (http://anf.ucsd.edu) displays a variety of diverse meta-data associated with the USArray Transportable and Flexible Arrays. A suite of online tools has been developed to allow visual and interactive exploration of the data and meta-data from the array stations. The website is partitioned into public and administrative areas. The public area provides dynamic maps of station locations, along with network affiliations, communication providers, hardware configurations, associated latencies and event maps. The public pages also show waveform data returned for the last 1, 2 and 24 hours. This allows rapid assessment of array health from anywhere with internet access. Graphical representations of state-of-health parameters are generated nightly for each station, creating a searchable archive. The station maps and metadata details are fully dynamic, updating immediately based on changes in the underlying databases and streaming data returned. The password-protected administrative area provides access to a searchable database of station-maintenance email from field personnel, filed by station, as well as upload and search facilities for digital photos of field sites integrated with the Flickr photo-sharing facility at http://www.flickr.com. The underlying architecture of this dynamic environment is the Datascope RDBMS, part of the Antelope Environmental Monitoring System (http://www.brtt.com), in combination with the PHP Hypertext Processing (PHP) scripting language. This configuration provides an easily extensible platform for real time environmental monitoring, with many re-usable components. We present the various types of data available from the ANF website, in addition to simple descriptions of how these data were collected and displayed. We also outline upcoming tools that we are currently developing to improve monitoring as this already large array continues to grow.

Large-scale observing systems are poised to become the dominant means of study for a variety of natural phenomena. These systems are comprised of hundreds to thousands of instruments that must be queried, managed, and shared in a scalable fashion. Services-oriented architectures (SOAs) are widely recognized as the preferred framework for building scalable and extensible cyberinfrastructure. By applying SOA concepts, we created a framework for organizing observing system resources. Guided by this framework, we developed web services, custom workflow applications, and an integrated user interface of monitors and controls for managing instruments in large-scale sensor network observing systems. In this paper we present our approach and discuss its application to the NSF EarthScope USArray large-scale seismic observing system.

EarthScope's Transportable Array has installed more than 200 high-quality broadband seismic stations over the last 3 years in the western US. These stations have a nominal spacing of 70 km and are part of an eventual 400 station array that migrates from west to east at a rate of 18 stations per month. The full 400 stations will be operating by September 2007. Stations have a residence time of about 2 years before being relocated to the next site. Throughout the continental US, 1623 sites are expected to be occupied. Standardized procedures and protocols have been developed to streamline all aspects of Transportable Array operations, from siting to site construction and installation to equipment purchasing and data archiving. Earned Value Management tools keep facility installation and operation on budget and schedule. A diverse, yet efficient, infrastructure installs and maintains the Transportable Array. Sensors, dataloggers, and other equipment are received and tested by the IRIS PASSCAL Instrument Center and shipped to regional storage facilities. To engage future geoscientists in the project, students are trained to conduct field and analytical reconnaissance to identify suitable seismic station sites. Contract personnel are used for site verification; vault construction; and installation of sensors, power, and communications systems. IRIS staff manages permitting, landowner communications, and station operations and maintenance. Seismic signal quality and metadata are quality-checked at the Array Network Facility at the University of California-San Diego and simultaneously archived at the IRIS Data Management Center in Seattle. Station equipment has been specifically designed for low power, remote, unattended operation and uses diverse two-way IP communications for real-time transmission. Digital cellular services, VSAT satellite, and commercial DSL, cable or wireless transport services are employed. Automatic monitoring of status, signal quality and earthquake event detection as well as operational alarms for low voltage and water intrusion are performed by a robust data acquisition package. This software is coupled with a host of network management tools and display managers operated by the Array Network Facility to allow managers, field personnel, and network operations staff to visualize array performance in real-time and to access historical information for diagnostics. Current data recording proficiency is 99.1%, with real-time telemetry averaging about 91%. EarthScope, IRIS and the USGS are working with regional seismic network operators, both existing and newly formed, to transition some of the Transportable Array stations into regional network assets. Each region has unique circumstances and interested parties are invited to exchange ideas on how this might be accomplished in their area. Contact busby@iris.edu for more information.

Recent progress on the Antelope interfaces to the Storage Resource Broker and on the deployable ROADNet Point Of Presence ("RPOP") has allowed three real-time observatories to be federated for virtual access to ROADNet image and meteorological data, coastal ocean current-monitoring data, and hundreds of gigabytes of USArray seismic data. The new Antelope-SRB Strdisp utility allows interactive review of waveform data from these data sources, with observatory-grade tools, on any computer with internet access, sufficient bandwidth, and system authentication. Similarly, the new Antelope-SRB Strexcerpt utility allows arbitrary subset databases of waveform data to be extracted, via simple or complex queries, and downloaded to the researcher's machine in any of approximately 20 user-specifiable formats. Key parts of the Antelope interface for PHP have been extended to the Antelope-SRB interface, providing web-scriptable world-wide-web access to these datasets. Similarly, key parts of the Antelope Toolbox for Matlab have been ported to the Antelope-SRB interface, making these large, federated data sets accessible to researchers for custom interactive exploration and processing based on their individual research needs. We end by discussing ongoing work on generalized remote database exploration and analysis tools for Antelope-SRB virtualized data sets. We also discuss the Antelope-SRB ORB (Virtual-ORB) interfaces in the RPOP cyberinfrastructure node.

The Antelope Environmental Monitoring System (http://www.brtt.com) is a robust middleware architecture for near-real-time data collection, analysis, archiving and distribution. Antelope has an extensive toolkit allowing users to interact directly with their datasets. A rudimentary interface was developed in previous work between Antelope and the web-scripting language PHP (The PHP language is described in more detail at http://www.php.net). This interface allowed basic application development for remote access to and interaction with near-real-time data through a World Wide Web interface. We have added over 70 new functions for the Antelope interface to PHP, providing a solid base for web-scripting of near-real-time Antelope database applications. In addition, we have designed a new structure for web sites to be created from the Antelope platform, including PHP applications and Perl CGI scripts as well as static pages. Finally we have constructed the first version of the dbwebproject program, designed to dynamically create and maintain web-sites from specified recipes. These tools have already proven valuable for the creation of web tools for the dissemination of and interaction with near-real-time data streams from multi-signal-domain real-time sensor networks. We discuss current and future directions of this work in the context of the ROADNet project. Examples and applications of these core tools are elaborated in a companion presentation in this session (Newman et al., AGU 2005, session IN06).

A 16.4 megapixel display wall has been constructed at the Array Network Facility (ANF) in the Cecil H. & Ida M. Green Institute of Geophysics and Planetary Physics (IGPP) for the display of multiple interactive visualizations of various geophysical datasets. This system is designed through collaboration between major NSF funded projects such as OptIPuter and USArray (Earthscope), and will allow researchers to access large stores of data on remote servers using dedicated links, visually analyze data and present results at extremely high resolution. The system is nicknamed iCluster and will be scaled to a 50 megapixel display as the USArray (currently about 80 stations) expands its operations. The OptIPuter project (www.optiputer.net) leverages the capabilities of dedicated optical networks that interconnect instruments, processors, computer storage and visualization resources to aid in Earth Sciences research. The iCluster comprises a cluster of three Apple Power Mac G5 machines and four Apple 30" LCD screens (of maximum resolution 2560 x 1600 each) tiled to form a 2x2 array and is the first tiled display built using Apple hardware (to our knowledge). The Antelope software is used for seismic data monitoring and archiving along with web-based analytical tools developed at the ANF (http://anf.ucsd.edu/). OptIPuter visualization software (developed by the Electronic Visualization Laboratory, University of Illinois at Chicago) such as JuxtaView (an image viewer for interacting with remotely located extremely high resolution 2D images) and Vol-a-Tile (interactive volume rendering software allowing navigation into gigabyte-sized seismic volumes) will also be used. Interactive visualizations created by scientists at IGPP that overlay heterogeneous datasets such as seismic profiles, geology strata, earthquake locations, bathymetry and high resolution satellite imagery and aerial photos, using the Fledermaus software can also be viewed. The configuration of each cluster node is: dual CPU 2.5 GHz PowerPC G5, 8 GB RAM, 500 GB disk space, NVIDIA Ultra 6800 GeForce card, and the nodes are interconnected over gigabit Ethernet. This system will also be part of the OptIPuter infrastructure, with fiber connections to the OptIPuter CAVEwave on the National Lambda Rail. The design of the system is based on the Geowall-2 class of displays and the OptIPuter Visualization Cluster at Scripps. More information about this system is available at (http://www.siovizcenter.ucsd.edu/icluster/index.html).

For anyone who has managed a moderately complex buffered real-time data transport system, the need for reliable adaptive data transport is clear. The ROADNet VORBrouter system, an extension to the ROADNet data catalog system [AGU-2003, Dynamic Dataflow Topology Monitoring for Real-time Seismic Networks], allows dynamic routing of real-time seismic data from sensor to end-user. Traditional networks consist of a series of data buffer computers with data transport interconnections configured by hand. This allows for arbitrarily complex data networks, which can often exceed full comprehension by network administrators, sometimes resulting in data loops or accidental data cutoff. In order to manage data transport systems in the event of a network failure, a network administrator must be called upon to change the data transport paths and to recover the missing data. Using VORBrouter, administrators can sleep at night while still providing 7/24 uninterupted data streams at realistic cost. This software package uses information from the ROADNet data catalog system to route packets around failed link outages and to new consumers in real-time. Dynamic data routing protocols operating on top of the Antelope Data buffering layer allow authorized users to request data sets from their local buffer and to have them delivered from anywhere within the network of buffers. The VORBrouter software also allows for dynamic routing around network outages, and the elimination of duplicate data paths within the network, while maintaining the nearly lossless data transport features exhibited by the underlying Antelope system. We present the design of the VORBrouter system, its features, limitations and some future research directions.

A 50 megapixel display wall is under construction at the Cecil H. & Ida M. Green Institute of Geophysics and Planetary Physics (IGPP) for the display of multiple interactive 3D visualizations of various geophysical datasets. This system is designed through collaboration between major NSF funded projects such as OptIPuter and USArray (EarthScope), and will allow researchers to visually analyze data and present results at extremely high resolution. The OptIPuter project (www.optiputer.net) leverages the capabilities of dedicated optical networks that interconnect instruments, processors, computer storage and visualization resources to aid in Earth Sciences research. This system comprises a cluster of seven Apple Power Mac G5 machines and twelve Apple 30" LCD screens (of maximum resolution 2560 x 1600 each) tiled to form a 4x3 array and will be the first Apple-driven tiled display to our knowledge. The Antelope software will be used for seismic data monitoring and archiving along with web-based analytical tools developed at the Array Network Facility (ANF http://anf.ucsd.edu/) at IGPP. OptIPuter software (developed by the Electronic Visualization Laboratory) such as JuxtaView (an image viewer for interacting with remotely located extremely high resolution 2D images) and Vol-a-Tile (interactive volume rendering software allowing navigation into gigabyte-sized seismic volumes) will also be used. Interactive visualizations created by scientists at IGPP that overlay heterogeneous datasets such as seismic profiles, geology strata, earthquake locations, bathymetry and high resolution satellite imagery and aerial photos, using the Fledermaus software will also be viewed. The configuration of each cluster node is: dual CPU 2.5 GHz PowerPC G5, 8 GB RAM, 500 GB disk space, NVIDIA Ultra 6800 GeForce card, and the nodes are interconnected over gigabit Ethernet. This system will also be part of the OptIPuter infrastructure, with fiber connections to the OptIPuter CAVEwave on the National Lambda Rail. The design of the system is based on the Geowall-2 class of displays (http://www.geowall.org) and the OptIPuter Visualization Cluster at Scripps (http://www.siovizcenter.ucsd.edu/optiputer/index.html).