1. Introduction The Data Mover Challenge 2021/22 is a challenge organized by NSCC Singapore since 2018. The charter of the DMC is to build a platform and bring together experts from industry and academia to test their high performance data transfer solutions across the Data Transfer Nodes testbed set up in various countries connected by 100G and beyond international networks. The challenge focuses on optimizing point-to-point data transfers between sites – a crucial step forward in advancing research collaboration and sharing. Participants from all over the world will compete by deploying the best software tools on Data Transfer Nodes that are set up within existing international networks across the globe. 2. The Challenge DMC 2021 will run from 1 August to 31 October 2021. Participating teams will be given 5 days to deploy and run data transfer tests based on given scenarios: - 3 days to set up the software - 2 days to run the data transfer and demonstrate software to judges A video conference zoom session will be set up for participants to present their solution and results with interview from the judges prior to the data transfer demonstration test. 3. Network Topology 4. Data Mover Challenge Competition Schedule 7 teams will take part in the competition. And Raysync will run the data transfer tests from 27th Sep to 1st Oct. The winning team’s leader will be invited to attend SupercomputingAsia 2022 https://www.scasia.org/ in Mar 2021 in Singapore for an award presentation and solution showcase. 5. About Raysync Solution Basing upon UDP’s high-speed transfer protocol, Raysync breaks through the bottleneck of large file transfer which satisfies enterprises’ needs by building an intelligent transfer platform that processes and integrates massive data at an unprecedented speed. Our solutions enable you to send files of any size or format at full line speed, hundreds of times faster than FTP, while ensuring secure and reliable delivery. For more info and pricing packages, please visit www.raysync.io.
The ability to effectively use big data to gain value comes down to the ability of organizations to run analytical applications on the data, usually in the data lake. Assume that the challenges of capacity, speed, diversity, and accuracy are solved-measuring data readiness. The data is ready to pave the way for predictive analysis. Data readiness is built on the quality of the big data infrastructure used to support business and data science analysis applications. For example, any modern IT infrastructure must support data migration associated with technology upgrades, integrated systems, and applications, and can transform data into required formats and reliably integrate data into a data lake or enterprise data warehouse ability. Three big challenges facing big data technology So why do so many big data infrastructures collapse early in the implementation life cycle? All this goes back to the last of McKinsey’s big data offer in 2011: “As long as the right policies and driving factors are formulated”. Some reasons why big data projects cannot be started are as follows: 1. Lack of skills Despite the increase in machine learning, artificial intelligence, and applications that can run without humans, the imagination to drive big data projects and queries is still a data scientist. These "promoters" referred to by McKinsey represent skills that are in great demand on the market and are therefore rare. Big data technology continues to affect the recruitment market. In many cases, big data developers, engineers, and data scientists are all on-the-job, learning processes. Many high-tech companies are paying more and more attention to creating and training more data-related positions to use the principles of big data. It is estimated that by 2020, 2.7 million people will be engaged in data-related jobs, and 700,000 of them will be dedicated to big data science and analysis positions to highly competitive and expensive employees. 2. Cost The value of the big data analytics industry is nearly $125 billion, and it is only expected to grow. For big data implementation projects, this means expensive costs, including installation fees and regular subscription fees. Even with the advancement of technology and the reduction of barriers to entry, the initial cost of big data may make the project impossible. Investment may require traditional consulting, outsourcing analysis, internal staffing, and storage and analysis software tools and applications. Various cost models are either too expensive, or provide the functionality of the minimum viable product, and cannot provide any actual results. But first, a company that wants to properly implement big data must prioritize architecture and infrastructure. 3. Data integration and data ingestion Before performing big data analysis, data integration must be performed first, which means that various data need to be sourced, moved, transformed, and provisioned into big data storage applications using technologies that ensure security. Control during the entire process. Modern integration technologies that connect systems, applications, and cloud technologies can help organizations produce reliable data gateways to overcome data movement problems. Companies striving to modernize their systems and deploy strategies to integrate data from various sources should tend to adopt a B2B-led integration strategy that ultimately drives the development of partner ecosystems, applications, data storage, and big data analysis platforms To provide better business value.
Facing the challenges and threats related to the security of big data transmission, the industry has conducted targeted practices and investigations on security protection technologies. This article focuses on three aspects of the development of big data security technology: platform security, data security, and privacy protection. Technologies related to platform security, data security, and privacy protection are improving, allowing us to solve big data security issues and challenges. However, to respond to new methods of cyber attacks, protect new data applications, and meet increased privacy protection requirements, higher standards and functions will be required. In terms of platform technology, centralized security configuration management and security mechanism deployment can meet the security requirements of the current platform. However, vulnerability scanning and attack monitoring technologies for big data platforms are relatively weak. In terms of technologies for defending platforms from network attacks, current big data platforms still use traditional network security measures to defend against attacks. This is not enough for the big data environment. In the big data environment, the extensible defense boundary is vulnerable to attack methods that cover up the intrusion. Besides, the industry pays little attention to potential attack methods that may come from the big data platform itself. Once new vulnerabilities appear, the scope of the attack will be huge. In terms of data security, data security monitoring and anti-sabotage technologies are relatively mature, but data sharing security, unstructured database security protection, and data violation traceability technologies need to be improved. Currently, there are technical solutions for data leakage: technology can automatically identify sensitive data to prevent leakage; the introduction of artificial intelligence and machine learning makes the prevention of violations move toward intelligence; the development of database protection technology also provides a powerful way to prevent data leakage Guarantee. The ciphertext calculation technology and the data leakage tracking technology have not yet been developed to the extent that they can meet the needs of practical applications, and it is still difficult to solve the confidentiality assurance problem of data processing and the problems related to tracking data flow. Specifically, the ciphertext calculation technology is still in the theoretical stage, and the calculation efficiency does not meet the requirements of practical applications. Digital watermarking technology cannot meet the needs of large-scale and fast-updated big data applications; data lineage tracking technology requires further application testing and has not yet reached the mature stage of industrial applications. Digital watermarking technology cannot meet the needs of large-scale and fast-updated big data applications; data lineage tracking technology requires further application testing and has not yet reached the mature stage of industrial applications. Digital watermarking technology cannot meet the needs of large-scale and fast-updated big data applications; data lineage tracking technology requires further application testing and has not yet reached the mature stage of industrial applications. In terms of privacy protection, technological development clearly cannot meet the urgent need for privacy protection. The protection of personal information requires the establishment of a guarantee system based on legal, technical, and economic methods. Currently, the widespread use of data desensitization technology poses challenges to multi-source data aggregation and may lead to failure. So far, there are few practical application case studies for emerging technologies such as anonymization algorithms, and there are other common problems with this technology, such as low computational efficiency and high overhead. In terms of computing, continuous improvement is needed to meet the requirements of protecting privacy in a big data environment. As mentioned earlier, the conflict between big data applications and personal information protection is not just a technical issue. Without technical barriers, privacy protection still requires legislation, strong law enforcement, and regulations to collect personal information for big data applications. Establish a personal information protection system that includes government supervision, corporate responsibility, social supervision, and self-discipline of netizens.
Big data transfer is becoming a new driving force for economic and social development, and is increasingly affecting economic operations, lifestyles and national governance capabilities. The security of big data transfer has been improved to the level of national security. Based on the challenges and problems facing big data transfer security and the development of big data security technology, we put forward the following 5 opinions for the development of big data security technology. 1. Build an integrated big data security defense system from the perspective of an overall security Security is a prerequisite for development. It is necessary to comprehensively improve the security of big data security technology, and then establish a comprehensive three-dimensional defense system that runs through the cloud management of big data applications to meet the needs of both countries. Big data strategy and it's market application. First, it is necessary to establish a security protection system covering the entire data life cycle, from collection to transfer, storage, processing, sharing, and final destruction. It is necessary to fully utilize data source verification, encryption of large-scale data transfer, encrypted storage in non-relational databases, privacy protection, data transaction security, prevention of data leakage, traceability, data destruction, and other technologies. The second is to enhance the security defense capabilities of the big data platform itself. It should introduce authentication for users and components, fine-grained access control, security audits for data operations, data desensitization, and other such privacy protection mechanisms. It is necessary to prevent unauthorized access to the system and data leakage while increasing attention to the inherent security risks involved in the configuration and operation of big data platform components. It is necessary to enhance the ability to respond to emergency security incidents that occur on the platform. Finally, it uses big data analysis, artificial intelligence, and other technologies to automatically identify threats, prevent risks and track attacks, and transition from passive defense to active detection. Ultimately, the goal is to enhance the security of big data from the bottom up and enhance the ability to defend against unknown threats. 2. Starting from attack defense, strengthen the security protection of big data platforms Platform security is the cornerstone of big data system security. From an earlier analysis, we can see that the nature of cyberattacks against big data platforms is changing. Enterprises are facing increasingly serious security threats and challenges. Traditional defensive surveillance methods will find it difficult to keep up with this change in the threat landscape. In the future, research on the security technology of big data transfer platforms should not only solve operational security issues but also design innovative big data platform security protection systems to adapt to the changing nature of cyber attacks. In terms of security protection technology, both open source and commercial big data platforms are in a stage of rapid development. However, the cross-platform security mechanism still has shortcomings. At the same time, the development of new technologies and new applications will reveal platform security risks that are not yet known. These unknown risks require all parties in the industry to start from the offensive and defensive aspects, invest more in the security of the big data platform, and pay close attention to the development trend of big data network attacks and defense mechanisms. It is necessary to establish a defense system suitable for and build a more secure and reliable big data platform. 3. Use key links and technologies as breakthrough points to improve the data security technology system In the big data environment, data plays a value-added role, its application environment is becoming more and more complex, and all aspects of the data life cycle are facing new security requirements. Data collection and traceability have become prominent security risks, and cross-organizational data cooperation is extensive, leading to confidentiality protection requirements that trigger multi-source aggregate computing. At present, technologies such as sensitive data identification, data leakage protection, and database security protection are relatively mature, while confidentiality protection in multi-source computing, unstructured database security protection, data security early warning, emergency response, and traceability of data leakage incidents, Still relatively weak. Actively promote the development of industry-university-research integration, and accelerate the research and application of key technologies such as ciphertext calculations to improve computing efficiency. Enterprises should strengthen support for data collection, calculation, traceability, and other key links; strengthen data security monitoring, early warning, control, and emergency response capabilities; take data security key links and key technology research as a breakthrough; improve the big data security technology system; To promote the healthy development of the entire big data industry. 4. Strengthen the investment in the industrialization of privacy protection core technologies, while considering the two important priorities of data use and privacy protection In the big data application environment, data usage and privacy protection will naturally conflict. Homomorphic encryption, secure multi-party computing, and anonymization technologies can strike a balance between the two and are ideal technologies to solve the privacy challenges in the application of big data. The advancement of core privacy protection technologies will inevitably greatly promote the development of big data applications. Currently, the core problem of privacy protection technology is efficiency, and its problems include high computing costs, high storage requirements, and lack of evaluation standards. Some researches, in theory, have not been widely used in engineering practice. It is very difficult to deal with privacy security threats such as multiple data sources or statistics-based attacks. In the big data environment, personal privacy protection has become a topic of much concern, and with the increasing demand for privacy protection in the future, it will drive the development and industrial application of dedicated privacy protection technologies. To improve the level of privacy protection technology in the big data environment, we must encourage enterprises and scientific research institutions to study privacy protection algorithms such as homomorphic encryption and secure multi-party computing, and at the same time promote data desensitization, audit applications, and other technical methods. 5. Pay attention to the research and development of big data security review technology and build a third-party security review system At present, the state has formulated a series of major decision-making arrangements for big data security. The government promotes the deep integration of big data and the real economy and emphasizes the need to effectively protect national data security. The National Informatization Plan puts forward an implementation plan for the big data security project. It is foreseeable that the government's supervision of big data security will be further strengthened in the future, the legislative process related to data security will be further accelerated, big data security supervision measures and technical means will be further improved, and the disciplinary work of big data security supervision will be further strengthened.
2021-02-05Big data transfer
File sharing|teletransmission|TLS|media industry|transfer files|cross-border data transmission|file transfer|long distance transmission|video transmission|file transfer|data sync|synchronous transmission|small file transfer|Secure file transfer|Send Large Files|shared file|mft|sftp|ftps|File sharing|aes|Data Management|point to point transfer|Fast File Transfer|Managed File Transfer|File transfer services|File transfer server|Transfer file via email|Transfer solution|Oversized file transfer|File transfer software|file sync|File synchronization software|Big data transfer|Transfer tool|file transfer protocol|ftp|File synchronization|High-speed file transfer|High speed transmission|transfer software|SD-WAN|High-speed transmission|Telecommuting|Data exchange| Foreign trade|File management|cloud computing|Operational tools|Enterprise Network Disk|saas|Cloud storage|Secure transmission|network|Cache|socks5|Breakpoint renewal|aspera|High speed transmission protocol|Transmission encryption|High Availability|Transnational transmission|FTP transmission|File synchronous transfer|High speed data transmission|Enterprise file transfer software|Large file transfer software|Data transmission software|Cross border transmission|Transfer large files|file data|File share transfer|Accelerated transmission|Transnational file transfer|Remote large file transfer|High speed transmission|tcp|HTTP|AD|LDAP|data transmission|raysync transmission|raysync cloud|file transfer|Large file transfer|File management system|Large file transfer|raysync Software|raysync|Large file transfer solution|raysync cloud|File transfer solution|Cross border file transfer|Transnational transmission|transmit data|network disk|transmission system|Point to point transmission|Mass file transfer|data sync