Josephus - The Jewish War (Penguin Edition). Everyone should read Josephus at least once. He provides an important, Jewish perspective of the events of the first century CE as well as a reasonable overview of the preceding 200 years. Readers will understand the world Jesus and Paul much better as they learn all of the various events not recorded in the Bible. I recommend the penguin edition because it offers an accessible translation with some helpful footnotes.
The Apocrypha - As with Josephus, the historical information contained in the Apocrypha is invaluable. For instance, readers of the Bible turn from Malachi to Matthew and discover a very different world. Suddenly there are Pharisees and Sadducees, groups that did not exist in the time of the prophets. And how and when did the Romans come to occupy that part of the world? The answers are found in the books of the Apocrypha. In addition to historical books there is also some good wisdom literature and expansions of some of the Old Testament stories. If you are looking for a good guide to this literature I would recommend David deSilva's Introducing the Apocrypha (Baker, 2004).
Everett Ferguson's The Backgrounds of Early Christianity (Eerdmans, 3rd ed, 2003) has become the standard in textbooks that introduces readers to the world of the New Testament. It is a handy reference tool that provides informative essays on a variety of topics like, religion, philosophy, the Roman Empire, slavery, family, etc. It is usually priced around $25-$30.
The Dictionary of New Testament Backgrounds (IVP, 2000) is a good reference volume to have on your shelf. It has hundreds of articles on various topics and events that are important for understanding the matrix of the New Testament. And it is affordable too, only about $40.
A good volume to have as you read the New Testament is Craig Keener's IVP Biblical Background Commentary to the New Testament (IVP, 1994). Though not as comprehensive as the above sources, Keener's volume will help you become familiar with the type of information that you need to know and will point you in the right direction.
Web sites that are helpful include Early Christian Writings.com. This is an excellent site to access many of the non-canonical Christian (and otherwise) literature. A companion site is Early Jewish Writings.com which will have some overlap with Early Christian site, but also has a host of other links. In addition to these, there is is always Mark Goodacre's NTGateway.com. Although not a backgrounds site, it does contain multiple helpful links to aid in research, some of which are relevant to Bible backgrounds.
I hope this is helpful. If you have a favorite resource why not post it below along with a link if there is one.