My Journey with LLVM (GSoC'20 Final Evaluation)

My GSoC comes to an end and this is a report of my work during the last 3 months. My project is adding DWARF support to yaml2obj, especially yaml2elf. The original proposal is here.

Implementation Status

With the help of my mentor James and other community members, I was able to accomplish most of the milestones in my original proposal. Now, the usability of the tool has been improved a lot. Some outstanding features are listed below.

* The InitialLength fields of DWARF sections are replaced with Format and Length. At first, we have to hardcode the InitialLength field to instruct the tool to emit a proper DWARF64 or a DWARF32 section, e.g.,

## DWARF32 section.
InitialLength:
  TotalLength32: 0x1234

## DWARF64 section.
InitialLength:
  TotalLength32: 0xffffffff
  TotalLength64: 0x1234

Now, yaml2obj emits DWARF32 sections by default and the Length field can be omitted, yaml2obj will calculate it for us (Patches that address this issue: D82622, D85880, D81063, D84008, D86590, D84911).

## DWARF32 section.
## The Format and Length fields can be omitted.
## We don't need to care about them.

## DWARF64 section.
Format: DWARF64 ## We only need to specify the Format field.

* yaml2obj supports emitting multiple abbrev tables. yaml2obj only supported emitting a single abbrev table and multiple compilation units had to share the same abbrev table before D86194 and D83116. Now, yaml2obj is able to emit multiple abbrev tables and compilation units can be linked to any one of them. We add an optional field ID to abbrev tables and an optional field AbbrevTableID to compilation units. Compilation units can use AbbrevTableID to link the abbrev table with the same ID. However, the AbbrOffset field of compilation units which corresponds to the debug_abbrev_offset field still needs to be specified. If D86614 can be accepted in the future, we don’t need to calculate it and specify it any more!

debug_abbrev:
  - ID: 0
    Table:
      ...
  - ID: 1
    Table:
      ...
debug_info:
  - ...
    AbbrevTableID: 1 ## Reference the second abbrev table.
  - ...
    AbbrevTableID: 0 ## Reference the first abbrev table.

* More DWARF sections are supported. The debug_rnglists, debug_loclists, debug_addr and debug_str_offsets sections are newly supported in yaml2obj. Check out D83624, D84234, D81541 and D83853 for more information!

* The DWARF support is added to elf2yaml and improved in macho2yaml. At first, the output of macho2yaml is noisy. It dumps the DWARF sections twice, one in the Sections: entry and one in the DWARF: entry, e.g.,

## The content of the debug_str section is dumped twice!
Sections:
  - sectname: __debug_str
    ...
    content: 6D61696E00 ## "main\0"
DWARF:
  debug_str:
    - main

After D85506, if the DWARF parser fails to parse the DWARF sections into the DWARF: entry, obj2yaml will dump them as raw content sections, otherwise, they will be presented as structured DWARF sections in the DWARF: entry. Besides, D85094 adds DWARF support to elf2yaml. Although it only supports dumping the debug_aranges section, we can easily extend it in the future.

Unfinished Tasks

* Allow users to describe DIEs at a high level. In my original proposal, we plan to make yaml2obj support describing DIEs at a high level. However, yaml2obj didn’t support emitting multiple abbrev tables at that time and I spent some time on enabling it to emit multiple abbrev tables and link compilation units with them. I’m not going to leave the community and I will improve it in the future.

My username on Phabricator is @Higuoxing. Please feel free to ping me if you have trouble in or encountering bugs in crafting DWARF test cases in YAML. I’m very happy to help!

Acknowledgements

I would love to express my sincere gratitude to @jhenderson(James Henderson) for mentoring me during this project. Besides, I would like to thank @grimar(George Rimar), @MaskRay(Fangrui Song), @labath(Pavel Labath), @dblaikie(David Blaikie), @aprantl(Adrian Prantl) and @probinson(Paul Robinson) for reviewing my patches, patiently answering my questions and leaving comments to my proposal!