Skip to main content

Petabox

Internet Archive Petabox
Petabox

A few highlights from the Petabox storage system:

  • No Air Conditioning, instead use excess heat to help heat the building.
  • Raw Numbers as of December 2021:
    • 4 data centers, 745 nodes, 28,000 spinning disks
    • Wayback Machine: 57 PetaBytes
    • Books/Music/Video Collections: 42 PetaBytes
    • Unique data: 99 PetaBytes
    • Total used storage: 212 PetaBytes






The Previous Version



OVERVIEW

The PetaBox(tm), custom-designed by Internet Archive staff, was originally created to safely store and process one petabyte (a million gigabytes) of information. The goals and design points were:
* Low power: 6kW per rack, 60kW for the entire storage cluster
* High density: 100+ TB/rack
* Local computing to process the data (800 low-end PC's)
* Multi-OS possible, linux standard
* Co-location friendly
* Shipping container friendly: Able to be run in a 20' by 8' by 8' shipping container.
* Easy Maintenance: One system administrator per petabyte
* Software to automate full mirroring
* Easy to scale
* Inexpensive design
* Inexpensive storage
The Internet Archive data center now houses ~3PB of PetaBox storage technology and is expanding steadily.


PETABOX HISTORY 6/2004

* The first 100TB Rack operational in Amsterdam as of June 2004.
* The second 80TB rack is operational in San Francisco
* Internet Archives spins off PetaBox production to newly-formed Capricorn Technologies.
2004 - 2007
* Capricorn replicates the Internet Archive's successful deployment of the PetaBox for major academic institutions, digital preservationists, government agencies, HPC and major research sites, medical imaging providers, digital image repositories, storage outsourcing sites, and other enterprises around the globe.

New PostPetabox email rss RSS

Subject Poster Replies Date
Web archive is not available for all the time jsuzana320 1
   Re: Web archive is not available for all the time Reclezon 0
Problems with connection margotfq 1
   Re: Problems with connection jsuzana320 0
Throttled again. ratherendit 2
   Re: Throttled again. rilysh 0
   Re: Throttled again. Texasluva 0
503ing for the very last time... ratherendit 1
   Re: 503ing for the very last time... Jeff Kaplan 1
     Re: 503ing for the very last time... ratherendit 0
St. Louis Vacation Rentals JZ Vacation Rentals 0
Issue in url Isaiah C. Allen 2
   Ads Ramazan Sancar 1
     Slow Loading Speeds andersoncop85 0
   Re: Issue in url jsuzana320 1
     Re: Issue in url andersoncop85 0
PetaBox TimCA 0
Odd storage formating question Reclezon 2
   Re: Odd storage formating question Jeff Kaplan 2
     Re: Odd storage formating question SeaMac 0
about how much storage is ready for growth twisted whiskers 2
   Re: about how much storage is ready for growth Richard Wayne530 1
Bittorrent tracker Evergreen_street 0
"Welcome to US Petabox" michaelreuter1 1
   Re: 'Welcome to US Petabox' varenhizzle 1
Silly question Teutamos 1
   Re: Silly question Richard Wayne530 1
Yes VUTHANHTUNG 0
Internet Archive https://www.HomelandInsecurity.net 0
petabox os openprotegent project 0
Petabox software specifications? crd6 0
How much archive.org monthly operating cost? tamim17 2
   Re: How much archive.org monthly operating cost? spartos 1
     not much covidbloggermailnes 0
Periodic updates icebob99 2
   Re: Periodic updates Jeff Kaplan 0
   Re: Periodic updates emijrp 0
Amazing JayKae 0
Wow... makanal 1
   Re: Wow... Jeff Kaplan 0
2014 ... 1 petabyte in 8U Javik 0
Design update? ihtoit 3
   Re: Design update? Coderjo 0
   Re: Design update? Coderjo 0
   Re: Design update? GridEngine 0
How long does the data last? Seaware 1
   Re: How long does the data last? Coderjo 1
     Re: How long does the data last? Seaware 1
       Re: How long does the data last? Coderjo 0
http://www.capricorn-tech.com/ is now dead ... new home? Tyler 1
   Re: http://www.capricorn-tech.com/ is now dead ... new home? Hydriz 1
     Re: http://www.capricorn-tech.com/ is now dead ... new home? Coderjo 1
       Re: http://www.capricorn-tech.com/ is now dead ... new home? منصة شليلة 0
We have 2TB of data to upload Sheikh Naveed 2
   Re: We have 2TB of data to upload fL0per 0
   Re: We have 2TB of data to upload Stright 1
6th pair of racks go into service: over 2PB of data space used brewster 3
   Re: 6th pair of racks go into service: over 2PB of data space used baetis 1
   Re: 6th pair of racks go into service: over 2PB of data space used Sean_0000001 0
   Re: 6th pair of racks go into service: over 2PB of data space used Tyler 0
New pictures of the new petabox brewster 3
   Re: New pictures of the new petabox Tyler 0
   Re: New pictures of the new petabox Coderjo 1
     Re: New pictures of the new petabox GridEngine 2
       Re: New pictures of the new petabox Coderjo 0
   Re: New pictures of the new petabox emijrp 0
Current Raw Numbers Sean_0000001 1
   Re: Current Raw Numbers brewster 0
could MapReduce be applied? broffmann 1
   Re: could MapReduce be applied? Coderjo 0
CAD files for Petabox? deerpig 2
   Re: CAD files for Petabox? thattommyhall 1
     Re: CAD files for Petabox? deerpig 1
       Re: CAD files for Petabox? deerpig 0
   Re: CAD files for Petabox? baetis 0
Is the Archive site no longer acrhiving maxadds 2
   Re: Is the Archive site no longer acrhiving direwolf0701 0
   Re: Is the Archive site no longer acrhiving brewster 0
I assume this forum is now obsolete Javik 3
   Re: I assume this forum is now obsolete Coderjo 2
     Re: I assume this forum is now obsolete tracey pooh 0
     Re: I assume this forum is now obsolete jack liu 0
Visiting the installation glenn 0
Intel Atom based motherboards? TimSmall 1
   Re: Intel Atom based motherboards? NoiseCollector 0
VIA NAS 7800-15LST dunno 0
SuperMicro Cascaded 128/256/384-drive SAS/SATA chassis Javik 1
   Understanding all that technical blather :-) Javik 1
     Re: Understanding all that technical blather :-) elhoim 0
FYI, don't need more PCI slots, just SATA-II or SAS Javik 2
   24-drives on one SAS channel, from SuperMicro Javik 2
     Oh look, the first ever 2.5 inch SAS hard drive Javik 0
     Re: Make that --144-- 2.5" drives on one SAS channel Javik 0
   Re: FYI, don't need more PCI slots, just SATA-II or SAS Gobba 0
Huge storage = huge challenges? cchipmunks 0
hi karatekid 1
   Re: hi cosmicharlie 0
As many pci slots as possible? AWP 1
   Re: As many pci slots as possible? elhoim 0
all archive storage is on petabox now! tracey pooh 1
   Re: all archive storage is on petabox now! protobadger 0
powernode rack brewster 1
   Re: powernode rack Coderjo 1
     Re: powernode rack brewster 1
       Re: powernode rack Coderjo 1
         Re: powernode rack brewster 0
current configuration? fgleason 1
   Re: current configuration? fgleason 1
     Re: current configuration? jkrauska 0
Still sept / oct for LMA > petabox move? Tyler 0
software specs jer0mey 1
   Re: software specs billmoyer 1
     Re: software specs singod 0
Hitachi 500GB drives ngngokkiu 1
   Re: Hitachi 500GB drives Nethacker 0
hey. I built something like that sxpert 1
   Re: hey. I built something like that JohnSebastianBach 0
1PB and Counting CR Saikley 0
You guys still on track for LMA transfer to Petabox.. end of summer 05? Tyler 0
like my movie? snipey_2 0
18 PCI slots, 72 SATA drives per computer? Javik 2
   Re: 18 PCI slots, 72 SATA drives per computer? caiser 1
     Re: 18 PCI slots, 72 SATA drives per computer? Javik 1
       Re: 18 PCI slots, 72 SATA drives per computer? elhoim 1
Drive spin-down? Low-RPM drives? Power saving? Javik 1
   Re: Drive spin-down? Low-RPM drives? Power saving? CR Saikley 1
     Re: Drive spin-down? Low-RPM drives? Power saving? HDRJapan 0
Hitachi Case Study Brad Leblanc 0
higher speed node motherboard brewster 0
petabox case design zippy 1
   Re: petabox case design CR Saikley 0
via board being used brewster 0
Massive network storage 2 Vladovlado 0
Massive network storage Vladovlado 1
   Re: Massive network storage caiser 0
about 400TB of this design shipped to the Archive brewster 5
   Re: about 400TB of this design shipped to the Archive foundation 0
   Re: about 400TB of this design shipped to the Archive Curator at the Security Digest Archives 1
     Re: about 400TB of this design shipped to the Archive Brak 1
   Re: about 400TB of this design shipped to the Archive dunno 1
     Re: about 400TB of this design shipped to the Archive brewster 1
       Re: about 400TB of this design shipped to the Archive jko 1
         Re: about 400TB of this design shipped to the Archive James Day 3
           Re: about 400TB of this design shipped to the Archive dunno 1
             Re: about 400TB of this design shipped to the Archive James Day 0
           Re: about 400TB of this design shipped to the Archive indianews 0
           Re: about 400TB of this design shipped to the Archive viswiss 0
   Re: about 400TB of this design shipped to the Archive matt-genesi-usa 0
   Re: about 400TB of this design shipped to the Archive indianews 0
Serial console cables? garan 1
   Re: Serial console cables? brewster 2
     Re: Serial console cables? garan 0
     Re: Serial console cables? Casandro 1
       Re: Serial console cables? sxpert 0
Possible solution: Disk Array Rackmounts KirbyMeist 1
   Re: Possible solution: Disk Array Rackmounts brewster 1
     Re: Possible solution: Disk Array Rackmounts foundation 1
       Re: Possible solution: Disk Array Rackmounts HDRJapan 0
Filesystem KirbyMeist 2
   Re: Filesystem brewster 4
     Re: Filesystem grignak 0
     Re: Filesystem foundation 1
       Re: Filesystem brewster 1
         Re: Filesystem foundation 1
           Re: Filesystem brewster 1
             Re: Filesystem dunno 1
               Re: Filesystem foundation 1
     Re: Filesystem Plasma 000 0
my shot at massive network storage. dunno 3
   Re: my shot at massive network storage. dunno 1
   Re: my shot at massive network storage. andyj 1
     Re: my shot at massive network storage. foundation 1
       Re: my shot at massive network storage. angelbassmuffin 0
Bootstrapping from USB Youssef Eldakar 2
   Re: Bootstrapping from USB billmoyer 1
1U power supplies Jeff Anderson-Lee 1

View more forum posts