
==== Front
Data Brief
Data Brief
Data in Brief
2352-3409
Elsevier

S2352-3409(24)00838-2
10.1016/j.dib.2024.110874
110874
Data Article
Bangla news article dataset
Saad Asif Mohammed
Mahi Umme Niraj
Salim Md. Shahidul
Hossain Sk Imran imran@cse.kuet.ac.bd
@skimranrana
⁎
Khulna University of Engineering & Technology, Khulna 9203, Bangladesh
⁎ Corresponding author. imran@cse.kuet.ac.bd@skimranrana
24 8 2024
12 2024
24 8 2024
57 1108747 6 2024
2 8 2024
19 8 2024
© 2024 The Author(s)
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
In this research, we present an updated standard Bangla dataset based on gathered Bangla news articles. In total, more than 1.9 million articles from nine Bangla news websites were gathered; the selection process was led by a number of categories, including sports, economy, politics, local news, tech, tourism, entertainment, education, health, the arts, and many more. The dataset per newspaper contains varying attributes, such as title, content, time, tags, meta, category, etc. This dataset will enable data scientists to investigate and assess theories related to Bangla natural language processing. Furthermore, there is a greater chance that the dataset will be utilized for domain-specific large language models in the context of Bangladesh, and it may be used to develop deep learning and machine learning models that categorize articles according to subjects.

Keywords

Data analysis
Classification
Natural language processing
==== Body
pmcSpecifications TableSubject	Data Science	
Specific subject area	Machine learning, Deep learning, Natural language processing (NLP), Text classifications, Text summarization, Bangla dataset.	
Type of data	JSONL files	
Data collection	The dataset was acquired from nine Bangla news websites using web-scraping tools. The duration of news gathered, widely varied, as per the online news available for each newspaper. Therefore, the earliest news date varies from newspaper to newspaper depending on their online archive. The categories were of diverse ranges, including national, politics, international, sports, education, entertainment, health, science and technology, etc. The distribution of news for each category varies widely. For each newspaper, all possible data has been gathered as permissible, such as Date and Time, Category, Title, Content, Tags, Meta. Python is the main language used to collect the articles, with two main packages used: Requests [1] and BeautifulSoup [2].	
Data source location	The news sources are: ajkerpatrika.com, banglatribune.com, dailyinqilab.com, bangla.dhakatribune.com, ekattor.tv, ittefaq.com.bd, dailyjanakantha.com, mzamin.com, samakal.com.	
Data accessibility	Repository name: Zenodo
Data identification number: 10.5281/zenodo.11111869
Direct URL to data: https://zenodo.org/records/11111869
Instructions for accessing these data: The dataset is publicly accessible using the provided direct URL.	
Related research article	‘none’	

1 Value of the Data

• The largest Bangla news article dataset incorporates 19,27,229 articles from nine different newspapers annotated according to their topics.

• The data can be used to perform different NLP tasks, such as text classification, text generation, sentence similarity, and text summarization.

• The articles from the newspapers contain various types of attributes, commonly including title, content, and time. Additionally, meta, keywords are scraped from newspapers where they are available at that moment.

• Length of contents of newspapers varies from 0 to around 20,000–30,000 characters and no duplicate articles exist in the dataset.

• Articles are not heavily filtered to retain the original context. So, the data can be widely used for various NLP related activities

• This data will help researchers in developing SOTA language models.

2 Background

This manuscript outlines the development of a Bangla News Articles dataset, initiated to support advancements in natural language processing (NLP) and large language models (LLMs), specifically tailored to the linguistic context of Bangladesh. The dataset is designed to be a comprehensive resource that facilitates the exploration of Bengali semantics. The main motivation behind the work is to address the lack of a large, up-to-date Bengali news article dataset suitable for LLMs.

Data collection methodologies employed included web scraping and crawling, with the resultant data being stored in the JSONL file format to enhance flexibility, storage efficiency, and data retrieval ease. This methodological approach is reflective of strategies previously applied in the ANAD: Arabic News Article Dataset [3], which similarly sought to compile domain-specific data within a regional context.

The dataset aligns with international NLP research trajectories while addressing unique challenges inherent to the Bengali language, thus supporting further studies in areas such as part-of-speech (POS) tagging, named entity recognition (NER), parsing, and machine translation. The provision of a robust corpus of Bengali text through this dataset is intended to enable more nuanced linguistic analysis and foster the development of superior language processing tools, contributing significantly to the advancement of language technologies in Bangladesh.

Building on the foundational research encapsulated in the Potrika [4], which gathered raw data to form a balanced dataset in the Bengali language characterised by key features and factors, this study expands the dataset substantially. The current dataset includes approximately 39.567 million sentences and 593.954 million words, sourced from 1927,229 news articles, and encompasses about six attributes across a diverse range of categories, derived from nine different newspapers, with updates continuing until March. This extensive dataset supports various applications, such as generating keyword similarity and dissimilarity maps, text generation, part-of-speech tagging, named entity recognition, and facilitating question answering.

Conversely, the original Potrika dataset comprises roughly 12.57 million sentences and 185.51 million words, collected from 664,880 news articles. It compiles information from six newspapers and organises it into five distinct attributes and eight specific categories, with updates up to the year 2020. This comparison highlights the broader scope and more current data of the newer dataset, positioning it as a more expansive tool for advanced linguistic and computational analyses

3 Data Description

Our dataset is publicly available in a data repository [5] for non-commercial uses. The data is gathered from a total of nine newspaper websites and the dataset is organized into nine JSON list files. Each JSON list file contains articles collected from the respective newspaper and every sample has six attributes: Title, Category, Time, Content, Meta, and Tags (Table 1).Table 1 Comparative analysis of key dataset features in our dataset and Potrika [4].

Table 1Features	Our Dataset	Potrika	
Number of sentences	39.567 million	12.57 million	
Word counts	593.954 million	185.51 million	
Number of articles	1927,229	664,880	
Number of attributes	At most 6, ranging from at least 4	5	
Range of categories	Varies from newspaper to newspaper, mentioned at Table 3	Specifically, 8	
Number of newspapers	9 newspapers	6 newspapers	
News timespan	Well updated to the latest	Discontinued from 2020	
Use cases	keyword similarity and dissimilarity maps, text generation, part-of-speech tagging, named entity recognition, and question answering	single-label text classification, text summarization, named entity recognition, word embedding model generation, news analysis task, machine translation	

Fig. 1 shows a sample from our dataset. The selected data sources represent three main categories: TV news channel websites, electronic media websites, and newspaper websites. The number of categories varies from newspaper to newspaper, ranging from 14 to 144. Articles are collected according to the categories available on each of the websites, and each article is annotated according to the category in its news portal source. The articles are organized according to the website source; they are classified according to the category to which they belong. Table 2 contains the overall description of newspaper article distribution. The distribution of articles per category for each website is summarized in Table 3. All the data is open source and available online.Fig. 1 Data sample in JSON list format.

Fig 1

Table 2 Quantitative description of the dataset.

Table 2Website	Number of category	Number of articles	Data attributes	
ajkerpatrika.com [6]	93	179,884	Title, Category, Time, Content, Tags, Meta	
banglatribune.com [7]	144	216,077	Title, Category, Time, Content, Tags	
dailyinqilab.com [8]	26	84,823	Title, Category, Time, Content, Meta	
bangla.dhakatribune.com [9]	14	78,160	Title, Category, Time, Meta, Content, Tags	
ekattor.tv [10]	27	67,392	Title, Category, Time, Content, Tags	
ittefaq.com.bd [11]	24	661,894	Title, Category, Time, Content, Tags	
dailyjanakantha.com [12]	16	310,321	Title, Category, Time, Content	
mzamin.com [13]	28	102,163	Title, Category, Time, Content	
samakal.com [14]	–	226,515	Title, Time, Content, Tags	
		Total = 19,27,229		

Table 3 Statistical description of newspapers based on their category and number of contents.

Table 3Newspaper	Category (Number of contents per category)	
Ajker Patrika [6]	Print edition(52,850), All over the country: Dhaka(21,675), All over the country: Chittagong(11,152), National(8224), All over the country: Rajshahi(8111), Sports: Cricket(7838), All over the country: Khulna(5692), Politics(5164), Nationwide: Rangpur(4507), Sports: Football(4056), Nationwide: Mymensingh(3396), Nationwide: Barisal(3304), Nationwide: Sylhet(2787), World: Europe(2604), Video(2552), World: India(2290), Economy(1986), Education(1931), World: Asia(1885), Technology(1778), Economy: Corporate(1557), World: USA(1540), Entertainment: Movies(1432), World: Arab(1284), Entertainment: Bollywood(1237), Pictures(1190), Lifestyle(1157), World(1102), Environment(1060), Career(924), Opinion, Sub-Editor(775), Sports(717), Law(710), All Country(686), Health(678), Entertainment: Music(648), Islam(598), Analysis(589), World: Pakistan(481), Technology: Social Media(474), Entertainment: Television(436), World: Africa(426), Entertainment(388), Economy: Financial Institutions(373), World: China(367), Chat(360), Entertainment: Hollywood(288), Lifestyle: Travel(282), Sports: Other Sports(278), Opinion: Editorial(267), Lifestyle: Women(264), Health: Doctor Advice(259), Technology: Gadgets(257), Education: Campus(242), Entertainment: Serial(238), Science(216), Special Issue(210), Sports: Tennis(209), Entertainment: South Movies(170), Interview(154), Career: Career Tips(151), Jobs(129), Tips(111), Science: Research(107), Lifestyle: Recipes(98), Art-Literature: Poetry(94), Opinions(77), Factcheck: Countries(76), Economics: Stock Market(68), FactCheck: Know But Wrong(65), Health: Research(62), Art-Literature(61), FactCheck: Today's Fact(59), FactCheck: Foreign(57), Industry- Literature: Story(52), Art-Literature: Essay(47), Quiz Competition(45), Entertainment: Folk-Culture(38), Economics: New Enterprise(38), Art-Literature: Discussion(34), Health: Tips(30), Games: Free Hits(23), Art-Literary: Children's(21), Science: Discovery Stories(8), FactCheck(7), Print Edition: Games(7), Print Edition: Opinion(5), Science: Scientist(3), Latest(2), Print Edition: Chittagong(1), Art-Literature: Romance(1), Technology: Know How(1), Print Edition: Today's Capital(1).	
Bangla Tribune [7]	National(26,180), Country(24,863), International News(22,883), Other(19,349), Sports, Cricket(12,012), Entertainment(9643), Country: Dhaka(7934), Country: Chittagong(7716), Sports: Football (6808), Politics (6567), Country: Khulna (6551), Business (5854), Living (5509), Country: Rajshahi (5319), Sports (4944), Country: Rangpur Division (4753), Law & Crime(4548), My Campus(3758), Column(3616), IT(3323), Country: Sylhet(2637), Country: Barisal(2604), Country: Mymensingh(1365), International News: Middle East(1262), Politics: BNP(1256), International News: USA(1115), International News: Asia(1070), Education(986), Sports: Other Sports(972), Journey(942), International News: India(920), Politics: Awami League(746), Job News(547), Health(543), International News: Europe(463), International News: United Kingdom(420), Politics: Teams & Organizations(400), Sports: Hockey(333), Sports: Tennis(297), Literature: Special Issues(262), Business: Power & Energy(258), International News: Africa(228), Other: Municipal Elections(227), Literature: Essays/Essays(223), Sports: BPL 2016(208), Sports: Tigers in Sri Lanka(186), Sports: Tigers in New Zealand(177), International News: Other(160), International News: America(150), International News: Russia(146), International News: China(130), Politics: National Party(129), Youth(121), Other: Election(115), Literature: Series(108), Other: Expatriation(102), Journey: Travel-Tour(93), Business: Business News(91), Literature: Interviews(90), International News: Germany(87), Literature: New Book News(84), Literature(83), Travel: Aviation(81), Other: Interviews(80), Literature: Short Stories(78), Entertainment: Bollywood(76), Politics: Jamaat-e-Islami(71), Journey: Weird(70), Literature: Poetry(69), IT, Technology News(65), International News: Latin America(64), Sports: Tigers in India(58), Sports: Gazi Ashraf Lipo's Column(54), Literature: Events(53), Business: Budget(51), Business: Finance & Investment (49), Journey: Tourism News(45), Literature: Awards(43), Entertainment: Hollywood(40), Literature: Reviews(40), Entertainment: Music(40), Other: Mourning & Remembrance(40), Journey: Hotels-Resorts(38), Entertainment: Dhalwood(36), International News: Japan(35), Sports: BPL 2019(28), Entertainment: OTT & Television(25), Literature: Reader Comments(22), Business- Commerce: Industry & Commerce(16), Other: Dhaka Lit Fest - 2017(16), Entertainment: Stage & Exhibition(15), Business: Stock Market(15), Other: Nashik Election 2016(14), Research(12), IT: Tricks(12), Other: Victory Stories(11), International News: Telescope(10), IT: Texclusive(10), Literature: Speech(8), Other: Capital(7), Entertainment: Abroad (7), Lifestyle: News Corner(6), Column: Haroon ur Rashid(5), Column: Muhammad Zafar Iqbal(5), Other: Reactions(5), Literature: Tabloid(5), Entertainment: Tollywood(5), Business: Global Economy(5), Lifestyle: Entertainment(4), IT: Games(4), Lifestyle: Health Tips(4), Columns: Shubo Kibria(4), Lifestyle: Special Features(3), Entertainment: Cannes Festival(3), Entertainment: Movies(3), Column: Prabhas Amin(3), Entertainment: Know What(3), Column: Salek Uddin(3), Living: Tips(3), Literature: Charuivati(2), Column: Bibhuranjan Sarkar(2), Other: Two Years(2), Information Technology: New in Market(2), Entertainment: New Light(2), Lifestyle: Fashion(2), Lifestyle: Beauty(2), Column: Bakhtiar Uddin Chowdhury(2), Column: Anees Alamgir(2), Column: Taslima Nasreen(1), Living: Living(1), Others: A. League's 20th Conference(1), Column: Amin Al Rashid(1), Breaking News(1), Lifestyle: Food Blog(1), IT: Tech-Talk(1), Entertainment: Internet(1), Column: Tushar Abdullah(1), Entertainment: Third Eye(1), Entertainment: This Day(1), Column: Chiranjan Sarkar(1), Column: Mahmudur Rahman(1), Other: Survey(1), Column: Fazlul Bari(1).	
Daily Inqilab [8]	Bangladesh(21,170), National(19,014), International(18,492), Sports(7859), Metropolitan(4635), Entertainment(3960), Domestic(2793), Editorial(1428), Today's Newspaper(1226), Economy(743), Politics(684), Peace and Prosperity Islam(370), Health(304), Expatriate Life(277), Literature(268), Islamic World(267), Religion Philosophy(259), Islamic Life(255), Business -Business(215), Special Issue(170), Islamic Q&A(139), ICT & Career(116), Special Report(97), Bongo Entertainment(52), Lifestyle(24), Spotlight(6).	
	
Dhaka Tribune Bangla [9]	Bangladesh(46,544), International(9889), Entertainment(4813), Sports(4600), Politics(4199), Other(2454), Economy(2016), Feature(1969), Technology(927), Opinion(372), News circular(240), Dhaka Lit Fest(64), Jobs(49), Education(24).	
	
Ekattor TV [10]	Country Yoga(19,275), World Yoga(13,179), National(10,062), Other(7586), Politics(4751), Sports(3623), Finance(1973), Capital(1903), Ananda Yoga(1641), Education(813), Health(722), Techniyoga(381), Prabas(377), Ektar Journal(365), Ektar Manch(223), Jeevan-Jipan(196), Mat-dvimat(107), Ektar Sanghan(95), Sangabadyog(42)), Muktijudda daily(34), Chitra-Vichitra(22), Khelayoga(9), Bangabandhu daily(4), Career(4), Sheikh Hasina daily(2), Women(2), Video(1).	
	
Daily Ittefaq [11]	Other(209,985), Nationwide(126,546), World News(67,140), National(52,656), Sports(46,988), Entertainment(32,029), Capital(31,042), Politics(19,853), Tech(12,020), Commerce(11,445), Education(11,025), Features(10,987), Opinion(7933), Lifestyle(7331), Expatriation(4483), Campus(2217), Generation(2130), Distinctions(1350), Jobs(1226), Literature(1183), Health(1118), Religion(1016), Press Release(190), PSI(1).	
	
Jana Kantha [12]	National(134,991), Bangladesh(47,117), Sports(32,342), Foreign(30,274), Economy(22,369), Entertainment(16,860), Opinion(5958), Different News(5647), Education(5028), Lifestyle(2915), Science and Technology (2433), Literature (1411), Special Issue (1150), Health (899), Environment (542), Politics (385).	
	
Daily Manab Zamin [13]	Banglaland(30,887), Online(15,790), Sports(12,252), Homeland(9470), Entertainment(7246), Worldland(6453), Last Page(5013), First Page(4673), Politics(3401), Kolkata Kathakta (1127), Body and Mind(732), Finance(726), India(690), Football World Cup(681), Education(581), Featured Column(491), Miscellaneous(440), T20 World Cup(435), Rokmari(267), Mat-Matantar(237), Prabas(179), Audio News(129), Information Technology(123), Apan Aloya Manavzamin(49), Sixteen Anna(27), From Qatar(23), Eid Anand 2023(22), World Cup Magazine(19).	

Ajker Patrika [6] is a Bengali-language daily newspaper that is published in Dhaka, Bangladesh. The newspaper started publishing on June 27, 2021. As of September 2022, the circulation of Ajker Patrika was around 1,08,100. The title lengths of articles vary from 2 to 128 characters. The timeline of articles collected from this website begins on March 18, 2021, and ends on December 24, 2023. In terms of content characteristics, 26,122 is the maximum character count. The 'Tags' attribute is present in 99.39 % of articles, whereas the 'Meta' attribute is present in only 1.124 % of articles.

Bangla Tribune [7] is a news website in Bengali originating from Bangladesh that commenced operations on May 13, 2014. As of July 2021, it held the #16 position in Bangladesh and was ranked #3064 globally, according to Alexa. The platform covers an extensive array of subjects, spanning politics, business, entertainment, sports, and technology. Additionally, it hosts numerous columnists and opinion contributors who offer their insights on contemporary issues. The timeline of articles collected from this website begins on November 25, 2015, and ends on November 30, 2019. Article title length ranges from 2 to 110 characters; in some cases, some unusual and unknown combinations of characters exist. For content attributes, the highest character size is 1,10,197. However, some contents have a very low number of characters, around 0–10; they may have no valuable information. 17.85 % of articles have the ‘meta data’ attribute available.

Daily Inqilab [8], a significant Bengali-language newspaper, is distributed daily in Bangladesh from its headquarters in Dhaka. Established on June 4, 1986, it operates under the motto ‘Solely for the nation and its citizens’. Characters in titles range from 2 to 426. The timeline of articles collected from this website begins on February 12, 2023, and ends on March 19, 2024. The maximum character count for content is 30,298. The 'meta data' attribute is present in 99.885 % of articles.

The Dhaka Tribune Bangla [9], a leading Bengali-language daily in Bangladesh, is based in Dhaka. It is popular among young people, expats, and the diplomatic community, with a broad readership both locally and internationally. Known for its diverse op-eds by renowned contributors globally, it's the fastest-growing Bangla news outlet in Bangladesh, catering to the business sector, middle class, and educational institutions. The newspaper boasts several award-winning journalists. Length of title characters: 7–110. The timeline of articles collected from this website begins on May 12, 2018, and ends on March 12, 2024. The content attributeʼs maximum character length is 24,023. 43.554 % of the data has 'Tags'. The data on 90.285 % is ‘meta data’.

Ekattor TV [10], a Bangladeshi news channel broadcasting in Bengali, signifies 'seventy-one,' referencing the 1971 Liberation War. It began broadcasting on June 21, 2012, as the country's initial news-focused channel in full HD. In 2023, the fact-checking body Rumor Scanner ranked Ekattor TV as Bangladesh's fifth-largest source of fake news. The channel operates from its headquarters on Sohrawardi Avenue in Baridhara. The timeline of articles collected from this website begins on March 7, 2021, and ends on March 12, 2024. Maximum length for content characters: 29,129. Articles with “Tags” comprise 82.784 %.

Daily Ittefaq [11], a prominent Bengali-language newspaper in Bangladesh, has been in circulation since 1953, making it the nation's oldest and among the most widely read newspapers. Published in broadsheet format by the Ittefaq Group of Publications Limited, it started as a weekly publication before transitioning to a daily format on December 24, 1953. The newspaper provides a daily electronic edition in Bengali on its website and targets a younger demographic with an English version. Additionally, it features an e-paper for digital readers. The distribution of title character lengths ranges from 1 to 275. The timeline of articles collected from this website begins on January 2, 2019 and ends on October 22, 2023. The maximum character for a content attribute is 1,43,644. Articles with ‘Meta’ data comprise 12.71 %.

Daily Janakantha [12], also known as “Daily People's Voice” in Bengali, is a daily newspaper that comes out of Dhaka, Bangladesh. It falls under the ownership of the Janakantha Shilpa Paribar. The publication first made its appearance on February 21, 1993. Daily Janakantha is available in both printed and online formats. There is a distribution of title characters from 2 to 255. Articles were gathered between April 24, 2022, and May 30, 2019. The longest character in the content is 44,489.

Daily Manab Zamin [13], “People's Land” in Bengali, is Bangladesh's leading tabloid, boasting the highest circulation among Bengali tabloids globally, with 19 million monthly online page views. It attracts 1.6 million international visitors from 189 countries, making it a top Bengali digital publication. Ranked in the world's top 500 newspaper websites and the top 1 % globally, it's the only Bangladeshi paper linked with FIFA, UEFA, and the English Premier League and has partnered with Hollywood studios for film promotions. It also hosted a dedicated 2010 FIFA World Cup website, ranking 15th in Bangladesh and second among Bengali-language news sites globally. The “Title” attribute has between two and one hundred and fifty-seven characters. The collection of articles takes place between April 16, 2022, and March 22, 2024. The maximum character count for content is 39,759.

Samakal [14], meaning “contemporary” in Bengali, is a daily newspaper published in Dhaka, Bangladesh. It was first published in 2005. With a circulation of 270,000 in 2020, it ranked as the eighth most widely read newspaper in Bangladesh. Throughout its existence, notable writers such as Ajoy Dasgupta, Abed Khan, Mizanur Rahman Khan, and Khalil Rahman have contributed to its pages. “Meta data” is included in 88.27 % of articles. The content attribute's maximum length is 42,470. The title may contain up to 106 characters. The period of time spans from September 30, 2019, to March 22, 2024. However, Samakal lacks a “Category” feature, and the chosen process of scraping from the website prevents us from retrieving the correct category value for any news. As a result, only “Title”, “Time”, “Content”, and “Tags” are properly recorded and arranged, and these attributes can be used for information mining, question answering, and sentence production in Bengali.

4 Experimental Design, Materials and Methods

The dataset included articles up to the possible latest period for all nine newspapers during the data collection process employing online-scraping tools and Python, which has a large number of libraries that facilitate web data retrieval, such as BeautifulSoup and Requests. One essential module for handling HTTP pages in Python is called Requests. The HTML pages from the cited sources were obtained using the Requests library. Furthermore, the article title and article body from HTML pages were extracted using BeautifulSoup.

In the process of gathering, the following criteria were taken into account in order to get articles from newspaper websites:• Each online page of a given newspaper follows a structural format. Thus, utilizing a loop assists in extracting information in a procedural manner.

• Extract the article's category, title, content, time, meta, and keywords; if possible to be obtained.

• The news website must be retrievable, as some websites are strictly unretrievable.

The proposed news article scraping algorithm is shown in Algorithm 1. The notations and inputs of the proposed algorithm are listed at the beginning of Algorithm 1. The algorithm is designed for efficient scraping of news articles, systematically retrieving, parsing, and storing article data from consecutive pages. Initially, in Step 1, it configures the necessary settings by loading the last scraped page index and setting paths for output files where the data will be logged and stored. This setup ensures a seamless continuation of data extraction across sessions. In step 2, the main scraping loop begins, continuously operating until manually stopped. During step 3, URLs are dynamically constructed for each successive page by appending the current index to the base URL. The algorithm then attempts to fetch the page content, and if the Step 4 HTTP response check is successful (i.e., status code 200), it proceeds to parse the HTML content in Step 4.1. From the parsed HTML, Step 4.2 involves extracting data elements like the article's title, publication time, main content, category, and associated keywords. Step 4.3 is only executed if both the title and content are non-empty, where it opens files for data logging and begins appending and formatting each piece of extracted information. Keywords and metadata are specifically processed if they exist, ensuring all relevant information is captured. In step 5, the algorithm updates the index to prepare for the next page and records this new index in a file to maintain progress. Finally, Step 6 ensures the algorithm continues to loop through the remaining pages, scraping data in a consistent and orderly way until the process is manually terminated. This structured approach ensures that the algorithm not only collects all available news data but also maintains a structured repository. As we collected data from nine newspapers, each with a unique and often changing website structure. To overcome this, we added a sample scraping code and tested it on a selection of articles from each year of every newspaper. We continuously monitored the collected data for errors, inconsistencies, and missing information, adjusting the code as needed after every ten thousand articles. This approach helped ensure data quality and consistency. The primary program operated autonomously, continuously scraping data from online sources, organizing it, and saving it in the specified format. Before final storage, the collected data underwent rigorous quality checks for errors, missing information, and inconsistencies. If any problems were detected, the main program was modified to rectify the data collection process. The source code is publicly available in the “source code.zip” file of our dataset [5].Algorithm 1 News article scraping algorithm.

Algorithm 1:Input: Base URL, Output Paths, Index File Path			
Base URL	–	The root URL to initiate scraping.	
Output Paths	–	Paths for the output text file and JSON lines file .	
Index File Path	–	Path to the file storing the last processed index.	
Output: An updated JSON lines file and a text file containing structured and logged data from scraped news articles.			
Variables:			
titleFinal	–	Holds the processed article title.	
categoryFinal	–	Holds the processed article category.	
timeFinal	–	Holds the processed article publication time.	
contentFinal	–	Holds the processed article content.	
keywordsFinal	–	Holds the processed article keywords.	
metaFinal	–	Holds the processed article metadata.	
outputText	–	Path to the file where scraped data is logged.	
lastvalText	–	Path to the file storing the last page index processed.	
jsonl	–	Path to the JSON lines file for structured data storage.	
currentIndex	–	Index for the page being processed	
httpResponse	–	Stores the HTTP response from the web request.	
parsedHtml	–	The parsed HTML document of the current page	
title	–	Holds the HTML element containing the news article's title.	
time	–	Stores the HTML element with the article's publication time.	
content	–	Stores the HTML element with the article's main text.	
meta	–	Stores the HTML element with the article's metadata or summary.	
category	–	Stores the HTML element with the article's category.	
keyword	–	Stores the HTML element with the article's associated keywords.	
raw_datasets	–	Structures scraped data.	
url	–	Combines the base URL with the current page index to access pages.	
1.		Step 1: Initialization	
2.		currentIndex ← Read from lastvalText.	
3.		outputText ← Paths for the output text file.	
4.		jsonl ← Paths for the JSON lines file.	
5.		//Initialize data variables	
6.		titleFinal ← “” // “” represents empty string	
7.		categoryFinal ← “”	
8.		timeFinal ← “”	
9.		contentFinal ← “”	
10.		keywordsFinal ← “”	
11.		metaFinal ← “”	
12.		raw_datasets ← Initialize with an empty structure prepared to store the dataset	
13.		Step 2: Main Scraping Loop	
14.		while True:	
15.		  Step 3: Construct URL and Fetch Page	
16.		  url ← “Base URL” + str(currentIndex)	
17.		  httpResponse ← requests.get(url)	
18.		  Step 4: Check HTTP Response	
19.		  if httpResponse.status_code = 200 then:	
20.		   Step 4.1 Parse HTML Content:	
21.		   parsedHtml ← Parse HTML from ‘httpResponse.text’	
22.		   Step 4.2 Extract Data Elements:	
23.		   title ← Extract main headline from the page.	
24.		   time ← Extract the publication date and time from the page.	
25.		   content ← Extract the main body text of the article.	
26.		   category ← Extract the articles's category or section.	
27.		   keyword ← Extract tags or keywords associated with the article.	
28.		   Step 4.3 Store Extracted Data:	
29.		   if title is not None and Content is not None then:	
30.		    Step 4.3.1 Open files for appending:	
31.		    file ← open file outputText with append mode and UTF-8 encoding.	
32.		    writer ← open JSON lines file at path jsonl with append mode.	
33.		    Step 4.3.2 Reset data storage variables	
34.		    Step 4.3.3 Store and append extracted data to text file	
35.		    Append contentFinal to file.	
36.		    Append currentIndex and titleFinal to file.	
37.		    Append timeFinal to file	
38.		    Step 4.3.4 Process and append keywords:	
39.		    if ‘keyword’ is not None then:	
40.		     keywordElements ← Get important words from ‘keyword’	
41.		     Initialize ‘keywordsFinal’ to “”	
42.		     for kw in keywordElements do:	
43.		      keywordsFinal ← Iterate and concatenate.	
44.		     end for	
45.		     write keywords to file	
46.		    end if	
47.		    Step 4.3.5 Process and append metadata:	
48.		     if meta is not None then:	
49.		      metaFinal ← Extract text within meta	
50.		     Step 4.3.6 Output Data to Files	
51.		    Save structured data to JSON lines file	
52.		    Save structured data to JSON lines file	
53.		  end if	
54.		  Step 5: Update Index and Save Progress	
55.		  currentIndex ← currentIndex + 1	
56.		  Update ‘lastvalText’ with new currentIndex	
57.		  Step 6: Loop Control	
58.		  The loop continues until manually stopped.	

We took into account certain undesired content throughout the retrieval process, including picture tags, additional HTML characters and tags, and other fast links after articles. Therefore, the following processing was conducted on each article to enhance the quality of the dataset:• A random search across all genres and publications was conducted to find any inappropriate content. For instance, over time, distinct HTML patterns have led to the identification of fast links in specific articles.

• Data attributes names were kept consistent across all newspapers.

• For keywords, they were concatenated and separated among them by comma (‘,’) sign.

• Meta data were kept as they were in the webpage, because, sometimes, they represent quotes or special verbal presentations in characters.

• In the content section, newlines were removed for better understanding, monitoring, and storage issues.

• For content, meta, keywords; NaN values have been replaced with empty string, which signifies missing data.

In the retrieval and filtering process, there were many challenges, as follows:• Some articles had very few content lengths. However, they were kept, as the data collection aim was to keep every possible data for future activities.

• Some articles had some storage issues, character related issues, duplication issues; all of them were removed from the database.

• Observing the HTML tags and identifying the required tags for different newspaper websites such as, the title, the body, timeframe, metadata, keywords, category of the article

• During the period of website scraping, the structure of websites changed and we needed to keep checking on retrieved data frequently.

We have validated our dataset using different classifiers. We have trained our dataset on support vector classification (SVC), k-nearest neighbours classifier, logistic regression, bagging classifier, perceptron, and gaussian naive bayes. Before training, we merged articles from semantically similar categories from each newspaper into a broad category. For example, for the Bangla Tribune newspaper ‘International News’, ‘India’, ‘Telescope’, ‘Asia’, ‘Other’, ‘Russia’, ‘Europe’, ‘International News, America’,‘USA’, ‘Middle East’, ‘Africa’, ‘Germany’, and ‘Latin America’ were merged to a category named ‘International’ whereas for the Ajker Patrika newspaper 'Abroad, world’, ‘Factcheck, abroad’, ‘Asia’, ‘India’, ‘USA’, ‘Europe’, ‘Africa’, ‘Arab’, and ‘Pakistan’ categories were merged to ‘International’ category. Table 4 contains details on the category merge operations for all nine newspapers, and Fig. 2 shows the data distribution of articles for each merged category. The category mapping is also available in the “category.txt” file in our data repository [5]. We merged the subcategories into main categories for easy visualization and classification. Fig. 3 shows the combined category-wise data distribution.Table 4 Category mapping (many to one) per category, per newspaper.

Table 4Newspaper	Merged category	Available categories	

Daily Inqilab [8]	‘Education’	‘Islamic Questions and Answers’, ‘Islam the Path to Peace and Prosperity’, ‘Islamic World’, ‘Philosophy of Religion’, ‘Islamic Life’	
‘Bangladesh’	‘National’, ‘Bangladesh’	
‘International’	‘International’	
‘Entertainment’	‘Bongo Entertainment’	
‘Information and Technology’	‘ICT and Career’	
‘Politics’	‘Politics’	
‘Economy’	‘Economy’	
‘Sports’	‘Sports’	

Bangla Tribune [9]	‘Education’	‘Education’, ‘My Campus’, ‘Job News’	
‘Bangladesh’	‘Country’, ‘Dhaka’, ‘Rajshahi’, ‘Sylhet’, ‘Khulna’, ‘Rangpur Division’, ‘Mymensingh’, ‘Chittagong’, ‘Barisal’	
‘International’	‘International News’, ‘India’, ‘Telescope’, ‘Asia’, ‘Other’, ‘Russia’, ‘Europe’, ‘International News, America’,
‘USA’, ‘Middle East’, ‘Africa’, ‘Germany’, ‘Latin America’	
‘Entertainment’	Bollywood’, New Light’, ‘Parvase’, ‘ Know What’, ‘Stage & Exhibition’, ‘Music’, ‘OTT & Television’, ‘Hollywood’, ‘Hollywood’, ‘Tollywood’, ‘Today’, ‘ Movies’, ‘Net’, ‘Cannes Festival’	
‘Information and Technology’	‘Tricks’, ‘Technology News’, ‘Games’, ‘Texclusive’, ‘New in the Market’, ‘Tech-Talk’	
‘Politics’	‘BNP’, ‘Municipal Elections’,
‘Stories of Victory’, ‘Awami League’, ‘Jamaat-e-Islami’, ‘Parties and Organizations’	
‘Economy’	‘Business’, ‘Finance & Investment’, ‘Stock Market’, ‘Business News’, ‘Power & Energy’	
‘Sports’	‘Sports’, ‘World Cup Magazine’, ‘Football World Cup’, ‘Cricket’, ‘Football’, ‘Other Sports’, ‘Tennis’, ‘ BPL 2016′, ‘Tigers in New Zealand’, ‘Tigers in India’, ‘Tigers in Sri Lanka’, ‘Khela, Gazi Ashraf Lipur Column’	

Ajker Patrika [6]	‘Education’	‘Campus’, ‘Career Tips’, ‘Career’	
‘Bangladesh’	‘Chittagong’, ‘Dhaka’, ‘Rajshahi’, ‘Khulna’, ‘Barisal’, ‘Rangpur’, ‘Sylhet’	
‘International’	‘Abroad,world’, ‘Factcheck, abroad’, ‘asia’, ‘India’, ‘USA’, ‘Europe’, ‘Africa’, ‘Arab’, ‘Pakistan’	
‘Entertainment’	‘Entertainment’, ‘Folk Culture’, ‘Bollywood’, ‘Television’, ‘Serial’, ‘Hollywood’, ‘South Cinema’	
‘Information and Technology’	‘Technology’, ‘Gadgets’, ‘Science’, ‘Discovery Stories’, ‘Research’	
‘Politics’	‘Politics’	
‘Economy’	‘Economy’, ‘Financial institution’	
‘Sports’	‘Tennis’, ‘Cricket’, ‘Other Sports’, ‘Football’, ‘Sports’	

Daily Ittefaq [11]	‘Education’	‘Education’	
‘Bangladesh’	‘National’, ‘Capital’, ‘Nationwide’	
‘International’	‘World news’	
‘Entertainment’	‘Entertainment’, ‘Joy’	
‘Information and Technology’	‘Tech’	
‘Politics’	‘Politics’	
‘Economy’	‘Trade’	
‘Sports’	‘Sports’	

Ekattor TV [10]	‘Education’	‘School’	
‘Bangladesh’	‘National’, ‘Country’, ‘Capital’	
‘International’	‘Global Communication’	
‘Entertainment’	‘Joy’	
‘Information and Technology’	‘Technology’	
‘Politics’	‘Politics’	
‘Economy’	‘Monetary’	
‘Sports’	‘Sports’	

Daily Manab Zamin [13]	‘Education’	‘School’	
‘Bangladesh’	‘Bangla’	
‘International’	‘World news’	
‘Entertainment’	‘Entertainment’	
‘Information and Technology’	‘Information Technology’	
‘Politics’	‘Politics’	
	‘Money – Trade’	
‘Sports’	‘Sports’, ‘World Cup Magazine’, ‘Football World Cup’	

Daily JanaKantha [12]	‘Education’	‘Education’	
‘Bangladesh’	‘Bangladesh’,’National’	
‘International’	‘Foreign’	
‘Entertainment’	‘Entertainment’	
‘Information and Technology’	‘Science and Technology’	
‘Politics’	‘Politics’	
‘Economy’	‘Economy’	
‘Sports’	‘Sports’	

DhakaTribune Bangla [9]	‘Education’	‘Education’	
‘Bangladesh’	'Bangladesh'	
‘International’	‘International’	
‘Entertainment’	‘Entertainment’	
‘Information and Technology’	‘Technology’	
‘Politics’	‘Politics’	
‘Economy’	‘Economy’	
‘Sports’	‘Sports’	

Fig. 2 Distribution of articles’ category per newspaper.

Fig 2

Fig. 3 Piechart showing data distribution for each category from all newspapers.

Fig 3

Based on Fig. 4, First, to use the Bangla dataset, access the data repository by visiting the provided URL [5] and download the necessary newspaper files. Use a programming language like Python to download the JSONL files and convert them into a suitable format such as CSV if required. Preprocess and clean the data by handling missing values, normalizing, tokenizing, and lemmatizing the text. Then visualize the data distribution and implement the text classification using machine learning models like Naive Bayes, SVM, or Deep Learning, and. Finally, evaluate the implementation by measuring performance metrics and interpret the results to refine the models.Fig. 4 Diagram for user guide on how to use dataset.

Fig 4

Limitations

This collection contains attributes and associated information from nine Bangladeshi newspapers. While collecting information and data from newspapers, we attempted to collect as much data as possible using our methods. However, unlike in other publications, the methodology used for “Samakal” did not allow us to extract the category attribute from the website. Similarly, some other newspapers' datasets are missing some attributes due to attribute unavailability on the website, as seen in Table 2. Some data properties could be accessed or automatically scraped on some websites, such as Daily Manab Zamin, while others remained inaccessible. Many websites do not support scripting, or their URLs lack adequate and easy indexing methods for crawling.

Ethics Statement

The authors have read and followed the ethical requirements for publication in Data in Brief and confirm that the work does not involve human subjects, animal experiments, or any data collected from social media platforms. The sources of news articles are public and available to anyone over the internet; thus, there would be no ethical questions regarding the storage, sharing, and utilization of the dataset. All the articles were acquired with the consent of the people, groups or organizations. The web resources allowed data to be scraped and distributed under their Terms of Service. We verified that our activities complied with these terms. The data sourced from the web belongs to the newspaper and is allowed for use under the website's copyright policies for non-commercial use. As it is public data and anyone can access it, there is no privacy concern for users. There are no scraping policies on those websites.

CRediT authorship contribution statement

Asif Mohammed Saad: Investigation, Methodology, Software, Data curation, Writing – original draft, Visualization. Umme Niraj Mahi: Investigation, Methodology, Software, Data curation, Writing – original draft, Visualization. Md. Shahidul Salim: Conceptualization, Methodology, Supervision, Validation, Writing – review & editing. Sk Imran Hossain: Conceptualization, Writing – review & editing.

Data Availability

BNAD: Bangla News Article Dataset (Original data) (Zenodo)

Acknowledgments

The work is supported by 10.13039/501100008677 Khulna University of Engineering & Technology.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could appear to influence the work reported in this paper.
==== Refs
References

1 K. Reitz, Requests: HTTP for Humans. https://requests.readthedocs.io/en/latest/, 2024 (accessed 1 April 2024).
2 L. Richardson, Beautiful Soup: a Python library for pulling data out of HTML and XML files. https://beautiful-soup-4.readthedocs.io/en/latest/, 2024(accessed 1 April 2024).
3 Altamimi M. Alayba A.M. ANAD: arabic news article dataset Data Br. 50 2023 109460 10.1016/j.dib.2023.109460
4 Ahmad I., AlQurashi F., M.R. Potrika: Raw and balanced newspaper datasets in the bangla language with eight topics and five attributes. arXiv preprint arXiv:2210.09389. 2022.
5 A.M. Saad, U.N. Mahi, M.S. Salim. (2024). BNAD: bangla News Article Dataset (Version V2) [Data set]. Zenodo. 10.5281/zenodo.11111869.
6 Ajker Patrika (Online newspaper). https://www.ajkerpatrika.com/, 2024 (accessed 1 April 2024).
7 Bangla Tribune (Online newspaper). https://www.banglatribune.com/, 2024 (accessed 27 March 2024).
8 Daily Inqilab (Online newspaper). https://dailyinqilab.com/, 2024 (accessed 27 March 2024).
9 Dhaka Tribune Bangla (Online newspaper). https://bangla.dhakatribune.com/, 2024 (accessed 12 March 2024).
10 Ekattor TV (Online news channel). https://ekattor.tv/, 2024 (accessed 12 March 2024).
11 Daily Ittefaq (Online newspaper). https://www.ittefaq.com.bd/, 2024 (accessed 20 March 2024).
12 Daily Janakantha (Online newspaper). https://www.dailyjanakantha.com/, 2024 (accessed 6 March 2024).
13 Daily Manab Zamin (Online newspaper). https://mzamin.com/, 2024 (accessed 23 March 2024).
14 Samakal (Online newspaper) https://samakal.com, 2024 (accessed 1 April 2024).
