Enhance Scraper with One-Time Date Range Functionality
- Added command line flags for specifying a date range and a one-time scraping mode in the scraper's main function. - Implemented validation for the date inputs to ensure both dates are provided and correctly formatted. - Introduced a new RunOneTimeScraping function to handle the one-time scraping logic, including context management and error handling. - Updated the TED scraper initialization to support both scheduled and one-time scraping modes, improving flexibility in data retrieval. - Enhanced logging to provide clear feedback on the scraping process and any errors encountered.
This commit is contained in:
+3
-1
@@ -45,7 +45,9 @@ func GetOJS(baseURL string, year int, date string) (string, error) {
|
||||
for _, row := range records[1:] {
|
||||
rowDate := strings.TrimSpace(row[1])
|
||||
if rowDate == strings.TrimSpace(date) {
|
||||
ojs = fmt.Sprintf("%v00%v", year, strings.TrimSpace(fmt.Sprint(row[0])))
|
||||
// Format: YEAR + zero-padded OJS number to make total length 9 digits
|
||||
// Examples: 202500004, 202500014, 202500196
|
||||
ojs = fmt.Sprintf("%d%05s", year, strings.TrimSpace(fmt.Sprint(row[0])))
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user