GMS Material Estimator — Eval Review

Skill 5 of 6 • Iteration 1 • 2026-03-28 • 21 assertions across 3 evals

With Skill
100%
21 / 21 assertions passed
Without Skill (Baseline)
19%
4 / 21 assertions passed
Delta
+81%
17 additional assertions passed

Eval 1: Quick Estimate — Pole Barn

With: 8/8 Without: 2/8
Prompt: I need a quick material estimate for a pole barn. 40 feet wide, 60 feet long, 14-foot eave height, 4:12 roof pitch. Classic Rib 29ga in Arctic White. Full metal walls on all sides. One 12x14 overhead door and one 3x7 walk door. Give me panel counts, trim, and fasteners.
AssertionTypeWith SkillWithout Skill
Correctly calculates roof area using 4:12 pitch multiplier (~1.05)CRITICAL PASS
2,400 sq ft footprint x 1.05 x 2 sides = 5,040 sq ft
 FAIL
Gets 2,820 sq ft using incorrect methodology — doesn't use 1.05 multiplier
Panel count uses 36" (3') coverage width, rounds UPCRITICAL PASS
ceil(60 / 3) = 20 panels/side, 40 total roof panels
 FAIL
Calculates 2,820 / (3 x 60) = 16 panels — wrong method, massive undercount
Includes ridge cap, eave trim, rake trim, drip edge, corner trim, and J-channel PASS
All 6 types plus Peak Cap and Rat Guard — 80 total pieces
 FAIL
Only ridge cap, eave/closure, corners. Missing rake, drip edge, J-channel
Subtracts door openings from wall panel counts PASS
Front: 20 - 4 = 16 panels. Side: 14 - 1 = 13 panels
 PASS
Subtracts 168 sq ft for overhead, 21 sq ft for walk door
Includes 5% waste on panels and 10% waste on trimCRITICAL PASS
Panels: 40 x 1.05 = 42. Trim: 80 x 1.10 = 88 pieces
 FAIL
Mentions "typically 5-10%" but never applies to actual counts
Fastener count uses ~80 screws/100 sq ft, rounds to boxes of 250 PASS
5,040 x 0.80 = 4,032 screws, ceil(4,032 / 250) = 17 boxes
 FAIL
"3,500 screws" with no per-sq-ft calc, no box count
Lists all assumptions explicitly PASS
Full ASSUMPTIONS section: gable roof, 12" overhang, exposed fastener, waste factors, supply-only
 FAIL
No explicit assumptions section — scattered notes only
Does NOT include installation labor or framingCRITICAL PASS
"NO labor, framing, or installation included — GMS supply only"
 PASS
No labor or installation mentioned

Eval 2: Detailed Estimate — Residential Re-Roof

With: 7/7 Without: 0/7
Prompt: I need a detailed material takeoff for a residential re-roof. The house is 30 feet wide by 50 feet long, 6:12 gable roof pitch, 12-inch overhangs on all sides. The homeowner wants SSQ550 standing seam in 26ga Kynar Slate Gray. There are 3 plumbing vent penetrations. No valleys. Include complete BOM with product codes.
AssertionTypeWith SkillWithout Skill
Uses SSQ550 coverage width (16" not 36")CRITICAL PASS
"SSQ550 has 16" coverage width (standing seam, not exposed fastener)"
 FAIL
Refused entirely — "I cannot provide an accurate material takeoff without access to your company's specific product data"
Applies 6:12 pitch multiplier (~1.12)CRITICAL PASS
1,500 x 1.12 x 2 sides = 3,360 sq ft
 FAIL
No calculation provided — full refusal
Calculates rafter length including 12" overhang using Pythagorean theorem PASS
sqrt(15^2 + 7.5^2) + 1' = 16.77' + 1' = 17.77'
 FAIL
Says "the calculation is straightforward" but doesn't do it
Includes 3 pipe boots for the plumbing vents PASS
3x Pipe Boot, 26ga, Kynar Slate Gray with product code PB26CMGSG
 FAIL
No material items listed at all
Product codes use CMG color format (CMGSG for Slate Gray) PASS
SSQ5506CMGSG, RC1026CMGSG10, RK26CMGSG10, DE26CMGSG10
 FAIL
No product codes — mentions "your product codes" generically
Includes standing seam clips instead of exposed fastener screws PASS
126x Standing Seam Clips, 24" OC — "no exposed roof screws"
 FAIL
No fastener specification of any kind
Mentions supply-only — homeowner needs installer PASS
"GMS supply-only: No framing, labor, or installation included"
 FAIL
Suggests querying JARVIS agents and Paradigm ERP instead

Eval 3: Quick vs Detailed Mode Awareness

With: 6/6 Without: 2/6
Prompt: A contractor calls and says 'I need to know roughly how much metal I need for a 30x40 garage with a 4:12 pitch. Classic Rib, 29ga, Charcoal. Full walls, one 16x8 overhead door.' Give me a quick estimate, then tell me what additional information you'd need for a detailed engineering-grade takeoff.
AssertionTypeWith SkillWithout Skill
Provides quick estimate FIRST, not questionsCRITICAL PASS
Starts with "QUICK ESTIMATE — 30x40 Garage" with full numbers
 PASS
Provides estimate first under "Quick Estimate" heading
Quick estimate includes panel counts, trim list, and fastener boxes PASS
28 roof panels, 74 wall. Trim: RC10(4), EVE(8), RK(7), DE(14), OC(4), JCH(4). Fasteners: 5+2 boxes
 FAIL
Gives LF of coil (2,450 LF) not panel counts. Trim is "100-150 LF" vague. No boxes
Lists detailed-mode questions: overhang, windows, penetrations, wainscot, soffit/fascia PASS
10 specific questions including wainscot, soffit & fascia, penetrations, overhang, gauge, color
 FAIL
Asks about coil width, stretchout, panel orientation — irrelevant. Missing wainscot, soffit/fascia
Uses correct panel coverage (36" for Classic Rib)CRITICAL PASS
ceil(40' / 3') = 14 panels per side for roof. Walls also use 3' coverage
 FAIL
Calculates in LF and sq ft, mentions "2-3 full coils" — not panel counts at 36" coverage
Subtracts 16x8 door opening from wall panels PASS
"Subtract 1 run for 16x8 overhead door: 18 panels front/back"
 PASS
"Less overhead door: 16' x 8' = 128 sq ft" subtracted from wall area
Includes waste factors in the quick estimate PASS
"WASTE INCLUDED: Panels: +5% built in, Trim: +10% built in"
 FAIL
No waste factors applied to any numbers
GMS Material Estimator • Skill 5 of 6 • Built with skill-creator • Greenfield Metal Sales • 2026