I have been doing my goal-setting, KPIs, and feedback assessments through LLMs. When HackerRank open-sourced their Hiring Agent, I was immediately drawn to it. I wanted to understand the rubric they use to evaluate candidates. Hiring Agent (https://github.com/interviewstreet/hiring-agent) is an LLM-based resume scoring tool that parses a PDF, enriches it with GitHub and blog data, and evaluates a score.

It’s straightforward to run - clone the repo, point it at a PDF:

$ git clone https://github.com/interviewstreet/hiring-agent
$ cd hiring-agent
$ python3 score.py /examples/java-engineer.pdf

The tool returns a structured score:

================================================================================
📊 RESUME EVALUATION RESULTS FOR: Senior Java Engineer
================================================================================

🎯 OVERALL SCORE: 71.0/100
🌐 Open Source:          12/35
🚀 Self Projects:        18/30
🏢 Production Experience: 25/25
💻 Technical Skills:      8/10

✅ KEY STRENGTHS:
------------------------------
  1. Microservices Architecture
  2. Event-Driven Systems
  3. Cloud Technologies
  4. Database Optimization
  5. API Design

🔧 AREAS FOR IMPROVEMENT:
------------------------------
  1. Lack of Open Source Contributions
  2. Limited GitHub Activity
  3. No Publicly Available Projects

Though the profile has a near-perfect score on production experience and technical skills, the tool still evaluated the candidate at 71. I traced the path to get a deeper understanding of how this tool works:

1. Resume Parsing

It starts in score.py where the PDF is handed off to a PDFHandler:

score.py:251-252

pdf_handler = PDFHandler()
resume_data = pdf_handler.extract_json_from_pdf(pdf_path)

Inside extract_json_from_pdf, the PDF is opened with PyMuPDF and converted to Markdown:

pdf.py:54-57

resume_text = to_markdown(
    doc,
    pages=pages,
)

The Markdown content is passed into structured JSON, an internal schema (basics, work, education, skills, projects, awards) that represents the content of the resume. The goal of this step is to normalise any resume format for reliable LLM evaluation. Now, parsing the resume from PDF to Markdown is not a foolproof step, in case of multi-column resumes direct Markdown conversion is unreliable.

Let’s take an example of this resume:

Resume evaluation pipeline output

The Markdown parser reads left-to-right, so the content is incorrectly extracted as:

Coding, Framework, Messaging, Database, Observability, Containers, Patterns, Cloud, Java, Python…

How well the LLM recovers from this depends on the model used. But the incorrect extraction does affect how the LLM evaluates the score. A more foolproof solution would be to screenshot the resume and let the model parse it directly, rather than converting to Markdown first.

2. Prompts and Rubric

~85% of the intelligence lives in prompts/templates/, not in .py files. evaluator.py is 90 lines; resume_evaluation_criteria.jinja alone is ~200 lines of scoring rules, fairness constraints, and anti-gaming patches.

The scoring weights:

open_source:       35 points
self_projects:     30 points
production:        25 points
technical_skills:  10 points

These numbers are the entire value system. Someone decided open source is 35% of an intern’s value.

The bonus structure:

+5  Google Summer of Code (GSoC)
+3  Girl Script Summer of Code
+3-5  startup founder/co-founder
+2-3  early-stage engineer
+2  portfolio website
+1  LinkedIn profile

Being a LinkedIn user is worth 1 point. Being a GSoC participant is worth 5. These are opinions stated as rules.

3. Fairness Engineering

What they guard against. The template explicitly forbids scoring based on name, gender, college, GPA, or location — stated in both the system message and the criteria template. GSoC and Girl Script Summer of Code are disambiguated by name (a real bug they hit and patched). Personal repositories are repeatedly distinguished from open source contributions. Hacktoberfest-only participation is capped at 3-5 points. These are battle scars, not theory.

What they don’t guard against. The weights themselves are a judgment call. Open source is 35% of an intern’s value — for someone who never touched GitHub, that’s a zero in a third of their score. Four commits is “minimal involvement” — but four commits to the right repo is real work. Tutorial projects trigger deductions — but a tutorial someone actually shipped is more than most people do. The fairness block prevents demographic bias. It doesn’t prevent value bias. That is the harder problem, and it is not solvable in a Jinja template.

A metric that looks objective but encodes someone’s opinion about what matters. Distorted Reality.

4. The Same Resume, Different Models

This tool manufactures a number that looks objective. The number depends on which model you pick, how the PDF converts, and what someone decided about open source being worth 35 points. That is not a criticism of the tool — it is the nature of turning judgment into a pipeline. The question is whether the people reading the number know that.

  1. The Pipeline
  2. The Product (Prompts)
  3. Fairness Engineering
  4. How This Affects Software Engineers and Market