In this article, we explore one of the key underlying mechanisms that mediates the human resource management (HRM)–performance link, namely, the (effective) HRM implementation by line managers. In particular, the purpose of our study is to compare middle and first-line managers’ experiences of their human resource (HR) role and the factors explaining effective HRM implementation at each managerial level. By employing survey data of two Belgian federal government organizations, we examine the effect of a number of organizational, individual and interpersonal factors on the effectiveness of line managers in HRM implementation. Results indicate that both middle and first-line managers’ HRM implementation effectiveness is related to organizational support, autonomous motivation, and coworker support. For the group of first-line managers, specifically, age and local office position are important in the execution of their HR tasks. For the group of middle managers, however, personnel red tape, length of service, and supervisory experience contribute to their HRM implementation effectiveness. As a result, an organization’s HR department can create the internal organizational conditions necessary to facilitate successful HRM implementation by the line. At the same time, it is worth adapting the approach according to the different managerial levels.