Week 09 Tutorial Questions

Objectives

  1. Write a Python program, tags.py which given the URL of a web page fetches it by running wget(1) and prints the HTML tags it uses.

    The tags should be converted to lower case and printed in alphabetical order with a count of how often each is used.

    Don't count closing tags.

    Make sure you don't print tags within HTML comments. 

        ./tags.py https://www.cse.unsw.edu.au
    a 141
    body 1
    br 14
    div 161
    em 3
    footer 1
    form 1
    h2 2
    h4 3
    h5 3
    head 1
    header 1
    hr 3
    html 1
    img 12
    input 5
    li 99
    link 3
    meta 4
    noscript 1
    p 18
    script 14
    small 3
    span 3
    strong 4
    title 1
    ul 25

    Note the counts in the above example will not be current - the CSE pages change almost daily.

  2. Add an -f option to tags.py which indicates the tags are to be printed in order of frequency. 

        ./tags.py -f https://www.cse.unsw.edu.au
    head 1
    noscript 1
    html 1
    form 1
    title 1
    footer 1
    header 1
    body 1
    h2 2
    hr 3
    h4 3
    span 3
    link 3
    small 3
    h5 3
    em 3
    meta 4
    strong 4
    input 5
    img 12
    br 14
    script 14
    p 18
    ul 25
    li 99
    a 141
    div 161

  3. Modify tags.py to use the requests and beautifulsoup4 modules.

  4. Write a Python program, emails.py which given the URL of a web page fetches it using curl via subprocess and prints all email addresses it finds.

    Print the addresses in alphabetical order, one per line, with duplicates removed.

    Use re.findall() with a regular expression to match email addresses in the form user@domain.tld. 

        ./emails.py https://www.unsw.edu.au/engineering/student-life/student-resources/key-contacts
    eng.curriculum@unsw.edu.au
    erikv@unsw.edu.au
    grievance-officer@cse.unsw.edu.au
    thesis-coordinator@cse.unsw.edu.au
    thesis.biomedeng@unsw.edu.au

  5. Write a Python program, dates.py which reads lines from standard input, and finds and stores the first date on each line. The program should then print the dates in sorted order, one per line.

    Support the following date formats:

    • DD/MM/YYYY (e.g. 25/12/2024),
    • MM/DD/YYYY (e.g. 12/25/2024),
    • YYYY-MM-DD (e.g. 2024-12-25),
    • DD Month YYYY (e.g. 25 December 2024), where Month is a full English month name.

    For each line, use a regular expression to find the date and convert it to the standard format YYYY-MM-DD. After reading all lines, print the converted dates in sorted order, one per line.

    When a date could be interpreted as either DD/MM/YYYY or MM/DD/YYYY (both parts ≤ 12), treat it as DD/MM/YYYY.

    If a line does not contain a date, ignore it. 

        ./dates.py
    25/12/2024
    12/25/2024
    2024-06-15
    1 January 2024
    not a date
    
    2024-01-01
    2024-06-15
    2024-12-25
    2024-12-25

  6. Write a Python program, headings.py which given the URL of a web page fetches it using the requests module, parses it with beautifulsoup4, and prints all heading text.

    Extract the text from <h1> through <h6> tags. Prepend each heading with indentation matching its level: one tab per level.

    Print the headings in the order they appear in the document.

    The beautifulsoup4 method get_text(strip=True) is useful for extracting clean text from tags. 

        ./headings.py https://www.cse.unsw.edu.au
        Follow
        Academic staff
        Professional staff
        Key contacts
        Contact us
        Research groups
        Facilities
        Research centres and institutes
        Research highlights
        Potential PhD projects
        Find a research supervisor
        Research focus areas
        John Lions Chair
        John Lions Distinguished Lecture Series
        Alumni profiles
        Industry Advisory Board
        Industry Partnership Program
        Industry Capstone Project
        Clubs & societies
        Prizes & awards
        Student projects
        CSE IT Helpdesk
        Enrolment sequences
        Course outlines
        Elite Students Program
        News
        Events
    The future of computing
            Redefining the human experience beyond the edge of possibility
        Latest news
            Women engineers honoured as UNSW awards celebrate excellence and Ada Lovelace Medal winner
            How artificial intelligence can best boost Australia’s future living standards
            Think online ads are harmless? They could be revealing your private life
            ‘Dangerously unprepared’: UNSW expert says Australia must regulate AI and invest in research
            Women engineers honoured as UNSW awards celebrate excellence and Ada Lovelace Medal winner
            How artificial intelligence can best boost Australia’s future living standards
            Think online ads are harmless? They could be revealing your private life
            ‘Dangerously unprepared’: UNSW expert says Australia must regulate AI and invest in research
        Our study areas
            Bioinformatics engineering
            Computer engineering
            Computer science and software engineering
            Data science
        Research & impact
            Research groups
            Research highlights
            Facilities
        Ways to connect
        Engage with us
        Study
        News, Media & Events
        About us
        Acknowledgement of Country
        Follow us

    The indentation above uses tabs (one per heading level) to show hierarchy.

  7. Modify headings.py to also count how many headings are at each level.

    Print a summary at the top of the output showing the count of each heading level in the format Hn: X where n is the level and X is the count. 

        ./headings.py https://www.cse.unsw.edu.au
    H1: 1
    H2: 37
    H3: 16
        Follow
        Academic staff
        Professional staff
        Key contacts
        Contact us
        Research groups
        Facilities
        Research centres and institutes
        Research highlights
        Potential PhD projects
        Find a research supervisor
        Research focus areas
        John Lions Chair
        John Lions Distinguished Lecture Series
        Alumni profiles
        Industry Advisory Board
        Industry Partnership Program
        Industry Capstone Project
        Clubs & societies
        Prizes & awards
        Student projects
        CSE IT Helpdesk
        Enrolment sequences
        Course outlines
        Elite Students Program
        News
        Events
    The future of computing
            Redefining the human experience beyond the edge of possibility
        Latest news
            Women engineers honoured as UNSW awards celebrate excellence and Ada Lovelace Medal winner
            How artificial intelligence can best boost Australia’s future living standards
            Think online ads are harmless? They could be revealing your private life
            ‘Dangerously unprepared’: UNSW expert says Australia must regulate AI and invest in research
            Women engineers honoured as UNSW awards celebrate excellence and Ada Lovelace Medal winner
            How artificial intelligence can best boost Australia’s future living standards
            Think online ads are harmless? They could be revealing your private life
            ‘Dangerously unprepared’: UNSW expert says Australia must regulate AI and invest in research
        Our study areas
            Bioinformatics engineering
            Computer engineering
            Computer science and software engineering
            Data science
        Research & impact
            Research groups
            Research highlights
            Facilities
        Ways to connect
        Engage with us
        Study
        News, Media & Events
        About us
        Acknowledgement of Country
        Follow us

  8. Modify headings.py to filter headings by level using a -l option.

    The -l option takes a level number (1-6) and only prints headings at that level or above (lower numbers are higher priority). For example, -l 3 would show <h1>, <h2>, and <h3> headings but not <h4>, <h5>, or <h6>. 

        ./headings.py -l 2 https://www.cse.unsw.edu.au
    H1: 1
    H2: 37
        Follow
        Academic staff
        Professional staff
        Key contacts
        Contact us
        Research groups
        Facilities
        Research centres and institutes
        Research highlights
        Potential PhD projects
        Find a research supervisor
        Research focus areas
        John Lions Chair
        John Lions Distinguished Lecture Series
        Alumni profiles
        Industry Advisory Board
        Industry Partnership Program
        Industry Capstone Project
        Clubs & societies
        Prizes & awards
        Student projects
        CSE IT Helpdesk
        Enrolment sequences
        Course outlines
        Elite Students Program
        News
        Events
    The future of computing
        Latest news
        Our study areas
        Research & impact
        Ways to connect
        Engage with us
        Study
        News, Media & Events
        About us
        Acknowledgement of Country
        Follow us

  9. If you feel like a harder challenge after finishing the challenge activity in the lab this week have a look at the following websites for some problems to solve using regular expressions: