Pigeonpea [Cajanus cajan (L.) Millsp.] is a versatile, stress-tolerant, and nutritious grain legume, possessing traits of value for enhancing the sustainability of dry sub-tropical and tropical agricultural systems. The use of crop wild relatives (CWR) in pigeonpea breeding has been successful in providing important resistance, quality, and breeding efficiency traits to the crop. Current breeding objectives for pigeonpea include increasing its tolerance to abiotic stresses, including heat, cold, drought, and waterlogging. Here we assess the potential for pigeonpea CWR to be further employed in crop improvement by compiling wild species occurrence and ex situ conservation information, producing geographic distribution models for the species, identifying gaps in the comprehensiveness of current germplasm collections, and using ecogeographic information to identify CWR populations with the potential to contribute agronomic traits of priority to breeders. The fifteen prioritized relatives of pigeonpea generally occur in South and Southeast Asia to Australia, with the highest concentrations of species in southern India and northern Australia. These taxa differ considerably among themselves and in comparison to the crop in their adaptations to temperature, precipitation and edaphic conditions. We find that these wild genetic resources are broadly under-represented in ex situ conservation systems, with 80% of species assessed as high priority for further collecting, thus their availability to plant breeders is insufficient. We identify species and highlight geographic locations for further collecting in order to improve the completeness of pigeonpea CWR germplasm collections, with particular emphasis on potential traits for abiotic stress tolerance.