The earliest segment of the Philip Tasker Building was constructed in the eighteenth century, possibly for a doctor. The house was built on the corner of Richmond Street and Fairfax Street. Fairfax Street led down to Bonner's Lane. The building is Grade II listed. In 1950 the listing document described it as red brick with ashlar (stone) dressings and a rainwater head dating to 1772. The interior is described as retaining some original details including the entrance hall with 2 segmental arches with fluted pilasters, the former dining room on the left has an original fireplace and an early 19th century triple arched buffet, while the room on the right has a grey marble fireplace, dentilated coving, moulded skirting and a dado rail. These rooms were carefully renovated by DMU in 2013 for use as function and reception rooms.
From October 1885 the building was used as the St Mary Magdalene Refuge for Fallen Women, sheltering and rehabilitating prostitutes drawn from any part of the Archdeaconry of Leicester. A separate wing of the building housed St Anne's Preventative Home meant to help vulnerable women before they fell into prostitution. A map of 1912, however, labels the building as an 'orphan asylum'. Some additions and extensions were made to the building in the 19th century to support these charitable functions.
The Refuge for Fallen Women was converted and extended into a school in 1931 by architect George Nott FRIBA, and was occupied by the Newarke Girls' School between 1932 and 1939. As well as a practicing architect, George Nott was also the Head of the Department of Architecture, Building and the Building Trades at the Leicester College of Art. The Gateway School for Boys moved into the Fairfax Street buildings in 1939 until 2009 when DMU took occupancy of the building.
Previously known as '1, Mill Lane', in March 2015 the building was named after Philip Tasker, scientist and Vice Chancellor of DMU from 1999 till 2010. The building is home to Leicester International Pathway College, Leicestershire Chamber of Commerce, the Centre for English Language Learning and the Sir Thomas White Charity.