Scrapy valueerror attempted relative import beyond top-level package python năm 2024

This article explores Python modules and Python packages, two mechanisms that facilitate modular programming.

Modular programming refers to the process of breaking a large, unwieldy programming task into separate, smaller, more manageable subtasks or modules. Individual modules can then be cobbled together like building blocks to create a larger application.

There are several advantages to modularizing code in a large application:

• Simplicity: Rather than focusing on the entire problem at hand, a module typically focuses on one relatively small portion of the problem. If you’re working on a single module, you’ll have a smaller problem domain to wrap your head around. This makes development easier and less error-prone.
• Maintainability: Modules are typically designed so that they enforce logical boundaries between different problem domains. If modules are written in a way that minimizes interdependency, there is decreased likelihood that modifications to a single module will have an impact on other parts of the program. (You may even be able to make changes to a module without having any knowledge of the application outside that module.) This makes it more viable for a team of many programmers to work collaboratively on a large application.
• Reusability: Functionality defined in a single module can be easily reused (through an appropriately defined interface) by other parts of the application. This eliminates the need to duplicate code.
• Scoping: Modules typically define a separate namespace, which helps avoid collisions between identifiers in different areas of a program. (One of the tenets in the Zen of Python is Namespaces are one honking great idea—let’s do more of those!)

Functions, modules and packages are all constructs in Python that promote code modularization.

Python Modules: Overview

There are actually three different ways to define a module in Python:

1. A module can be written in Python itself.
2. A module can be written in C and loaded dynamically at run-time, like the re (regular expression) module.
3. A built-in module is intrinsically contained in the interpreter, like the itertools module.

A module’s contents are accessed the same way in all three cases: with the import statement.

Here, the focus will mostly be on modules that are written in Python. The cool thing about modules written in Python is that they are exceedingly straightforward to build. All you need to do is create a file that contains legitimate Python code and then give the file a name with a .py extension. That’s it! No special syntax is necessary.

For example, suppose you have created a file called mod.py containing the following:

mod.py

Several objects are defined in mod.py:

• s (a string)
• a (a list)
• foo() (a function)
• Foo (a class)

Assuming mod.py is in an appropriate location, which you will learn more about shortly, these objects can be accessed by importing the module as follows:

The Module Search Path

Continuing with the above example, let’s take a look at what happens when Python executes the statement:

When the interpreter executes the above import statement, it searches for mod.py in a list of directories assembled from the following sources:

• The directory from which the input script was run or the current directory if the interpreter is being run interactively
• The list of directories contained in the environment variable, if it is set. (The format for `itertools`3 is OS-dependent but should mimic the `itertools`5 environment variable.)
• An installation-dependent list of directories configured at the time Python is installed

The resulting search path is accessible in the Python variable `itertools`6, which is obtained from a module named `itertools`7:

Thus, to ensure your module is found, you need to do one of the following:

• Put mod.py in the directory where the input script is located or the current directory, if interactive
• Modify the itertools`3 environment variable to contain the directory where `mod.py is located before starting the interpreter
  • Or: Put mod.py in one of the directories already contained in the `itertools`3 variable
• Put mod.py in one of the installation-dependent directories, which you may or may not have write-access to, depending on the OS

There is actually one additional option: you can put the module file in any directory of your choice and then modify itertools`6 at run-time so that it contains that directory. For example, in this case, you could put `mod.py in directory `import`6 and then issue the following statements:

Once a module has been imported, you can determine the location where it was found with the module’s `import`7 attribute:

The directory portion of `import`7 should be one of the directories in `itertools`6.

The import Statement

Module contents are made available to the caller with the import statement. The import statement takes many different forms, shown below.

`.py`3

The simplest form is the one already shown above:

Note that this does not make the module contents directly accessible to the caller. Each module has its own private symbol table, which serves as the global symbol table for all objects defined in the module. Thus, a module creates a separate namespace, as already noted.

The statement .py`3 only places .py`5 in the caller’s symbol table. The objects that are defined in the module remain in the module’s private symbol table.

From the caller, objects in the module are only accessible when prefixed with `.py`5 via dot notation, as illustrated below.

After the following import statement, .py`8 is placed into the local symbol table. Thus, .py`8 has meaning in the caller’s local context:

But s and `mod.py`1 remain in the module’s private symbol table and are not meaningful in the local context:

To be accessed in the local context, names of objects defined in the module must be prefixed by `.py`8:

Several comma-separated modules may be specified in a single import statement:

`mod.py`4

An alternate form of the import statement allows individual objects from the module to be imported directly into the caller’s symbol table:

Following execution of the above statement, mod.py`6 can be referenced in the caller’s environment without the .py`5 prefix:

Because this form of import places the object names directly into the caller’s symbol table, any objects that already exist with the same name will be overwritten:

It is even possible to indiscriminately import everything from a module at one fell swoop:

This will place the names of all objects from `.py`5 into the local symbol table, with the exception of any that begin with the underscore (`mod.py`1) character.

For example:

This isn’t necessarily recommended in large-scale production code. It’s a bit dangerous because you are entering names into the local symbol table en masse. Unless you know them all well and can be confident there won’t be a conflict, you have a decent chance of overwriting an existing name inadvertently. However, this syntax is quite handy when you are just mucking around with the interactive interpreter, for testing or discovery purposes, because it quickly gives you access to everything a module has to offer without a lot of typing.

`mod.py`2

It is also possible to import individual objects but enter them into the local symbol table with alternate names:

This makes it possible to place names directly into the local symbol table but avoid conflicts with previously existing names:

`mod.py`4

You can also import an entire module under an alternate name:

Module contents can be imported from within a function definition. In that case, the import does not occur until the function is called:

However, Python 3 does not allow the indiscriminate `mod.py`6 syntax from within a function:

Lastly, a mod.py`7 statement with an `mod.py`8 clause can be used to guard against unsuccessful `import attempts:

The `s`0 Function

The built-in function `s`0 returns a list of defined names in a namespace. Without arguments, it produces an alphabetically sorted list of names in the current local symbol table:

Note how the first call to `s`0 above lists several names that are automatically defined and already in the namespace when the interpreter starts. As new names are defined (`s`3, `s`4, `s`5), they appear on subsequent invocations of `s`0.

This can be useful for identifying what exactly has been added to the namespace by an import statement:

When given an argument that is the name of a module, `s`0 lists the names defined in the module:

Executing a Module as a Script

Any .py file that contains a module is essentially also a Python script, and there isn’t any reason it can’t be executed like one.

Here again is mod.py as it was defined above:

mod.py

This can be run as a script:

There are no errors, so it apparently worked. Granted, it’s not very interesting. As it is written, it only defines objects. It doesn’t do anything with them, and it doesn’t generate any output.

Let’s modify the above Python module so it does generate some output when run as a script:

mod.py

Now it should be a little more interesting:

Unfortunately, now it also generates output when imported as a module:

This is probably not what you want. It isn’t usual for a module to generate output when it is imported.

Wouldn’t it be nice if you could distinguish between when the file is loaded as a module and when it is run as a standalone script?

Ask and ye shall receive.

When a .py file is imported as a module, Python sets the special dunder variable `a`1 to the name of the module. However, if a file is run as a standalone script, `a`1 is (creatively) set to the string `a`3. Using this fact, you can discern which is the case at run-time and alter behavior accordingly:

mod.py

Now, if you run as a script, you get output:

But if you import as a module, you don’t:

Modules are often designed with the capability to run as a standalone script for purposes of testing the functionality that is contained within the module. This is referred to as unit testing. For example, suppose you have created a module `a`4 containing a factorial function, as follows:

fact.py

The file can be treated as a module, and the `a`5 function imported:

But it can also be run as a standalone by passing an integer argument on the command-line for testing:

Reloading a Module

For reasons of efficiency, a module is only loaded once per interpreter session. That is fine for function and class definitions, which typically make up the bulk of a module’s contents. But a module can contain executable statements as well, usually for initialization. Be aware that these statements will only be executed the first time a module is imported.

Consider the following file mod.py:

mod.py

The `a`7 statement is not executed on subsequent imports. (For that matter, neither is the assignment statement, but as the final display of the value of `a`8 shows, that doesn’t matter. Once the assignment is made, it sticks.)

If you make a change to a module and need to reload it, you need to either restart the interpreter or use a function called `a`9 from module `foo()`0:

Python Packages

Suppose you have developed a very large application that includes many modules. As the number of modules grows, it becomes difficult to keep track of them all if they are dumped into one location. This is particularly so if they have similar names or functionality. You might wish for a means of grouping and organizing them.

Packages allow for a hierarchical structuring of the module namespace using dot notation. In the same way that modules help avoid collisions between global variable names, packages help avoid collisions between module names.

Creating a package is quite straightforward, since it makes use of the operating system’s inherent hierarchical file structure. Consider the following arrangement:

Here, there is a directory named `foo()`1 that contains two modules, `foo()`2 and `foo()`3. The contents of the modules are:

mod1.py

mod2.py

Given this structure, if the `foo()`1 directory resides in a location where it can be found (in one of the directories contained in `itertools`6), you can refer to the two modules with dot notation (`foo()`6, `foo()`7) and import them with the syntax you are already familiar with:

You can import modules with these statements as well:

You can technically import the package as well:

But this is of little avail. Though this is, strictly speaking, a syntactically correct Python statement, it doesn’t do much of anything useful. In particular, it does not place any of the modules in `foo()`1 into the local namespace:

To actually import the modules or their contents, you need to use one of the forms shown above.

Package Initialization

If a file named `foo()`9 is present in a package directory, it is invoked when the package or a module in the package is imported. This can be used for execution of package initialization code, such as initialization of package-level data.

For example, consider the following `foo()`9 file:

__init__.py

Let’s add this file to the `foo()`1 directory from the above example:

Now when the package is imported, the global list `Foo`2 is initialized:

A module in the package can access the global variable by importing it in turn:

mod1.py

`foo()`9 can also be used to effect automatic importing of modules from a package. For example, earlier you saw that the statement `Foo`4 only places the name `foo()`1 in the caller’s local symbol table and doesn’t import any modules. But if `foo()`9 in the `foo()`1 directory contains the following:

__init__.py

then when you execute `Foo`4, modules `Foo`9 and `itertools`00 are imported automatically:

Importing `itertools`01 From a Package

For the purposes of the following discussion, the previously defined package is expanded to contain some additional modules:

There are now four modules defined in the `foo()`1 directory. Their contents are as shown below:

mod1.py

mod2.py

mod3.py

mod4.py

(Imaginative, aren’t they?)

You have already seen that when `mod.py`6 is used for a module, all objects from the module are imported into the local symbol table, except those whose names begin with an underscore, as always:

The analogous statement for a package is this:

What does that do?

Hmph. Not much. You might have expected (assuming you had any expectations at all) that Python would dive down into the package directory, find all the modules it could, and import them all. But as you can see, by default that is not what happens.

Instead, Python follows this convention: if the `foo()`9 file in the package directory contains a list named `itertools`05, it is taken to be a list of modules that should be imported when the statement `itertools`06 is encountered.

For the present example, suppose you create an `foo()`9 in the `foo()`1 directory like this:

pkg/__init__.py

Now `itertools`09 imports all four modules:

Using `mod.py`6 still isn’t considered terrific form, any more for packages than for modules. But this facility at least gives the creator of the package some control over what happens when `mod.py`6 is specified. (In fact, it provides the capability to disallow it entirely, simply by declining to define `itertools`05 at all. As you have seen, the default behavior for packages is to import nothing.)

By the way, `itertools`05 can be defined in a module as well and serves the same purpose: to control what is imported with `mod.py`6. For example, modify `foo()`2 as follows:

pkg/mod1.py

Now an `mod.py`6 statement from `foo()`6 will only import what is contained in `itertools`05:

foo() (the function) is now defined in the local namespace, but Foo (the class) is not, because the latter is not in `itertools`05.

In summary, `itertools`05 is used by both packages and modules to control what is imported when `mod.py`6 is specified. But the default behavior differs:

• For a package, when `itertools`05 is not defined, `mod.py`6 does not import anything.
• For a module, when `itertools`05 is not defined, `mod.py`6 imports everything (except—you guessed it—names starting with an underscore).

Subpackages

Packages can contain nested subpackages to arbitrary depth. For example, let’s make one more modification to the example package directory as follows:

The four modules (`foo()`2, `foo()`3, `itertools`30 and `itertools`31) are defined as previously. But now, instead of being lumped together into the `foo()`1 directory, they are split out into two subpackage directories, `itertools`33 and `itertools`34.

Importing still works the same as shown previously. Syntax is similar, but additional dot notation is used to separate package name from subpackage name:

In addition, a module in one subpackage can reference objects in a sibling subpackage (in the event that the sibling contains some functionality that you need). For example, suppose you want to import and execute function foo() (defined in module `Foo`9) from within module `itertools`37. You can either use an absolute import:

pkg/sub__pkg2/mod3.py

Or you can use a relative import, where `itertools`38 refers to the package one level up. From within `itertools`30, which is in subpackage `itertools`34,

• `itertools`38 evaluates to the parent package (`foo()`1), and
• `itertools`43 evaluates to subpackage `itertools`33 of the parent package.

pkg/sub__pkg2/mod3.py

Conclusion

In this tutorial, you covered the following topics:

• How to create a Python module
• Locations where the Python interpreter searches for a module
• How to obtain access to the objects defined in a module with the import statement
• How to create a module that is executable as a standalone script
• How to organize modules into packages and subpackages
• How to control package initialization

This will hopefully allow you to better understand how to gain access to the functionality available in the many third-party and built-in modules available in Python.

Additionally, if you are developing your own application, creating your own modules and packages will help you organize and modularize your code, which makes coding, maintenance, and debugging easier.

How do I fix relative import error in Python?

What you need to do is run the main file of your package in order for the import to work. In other words, don't open the urls.py and hit run (or whatever command your IDE has for running the module). Run the top-most main file of the relative package (i.e., your main file). If you do it this way, the import will work.

Should Python import be placed at the top of the module?

It is customary but not required to place all import statements at the beginning of a module (or script, for that matter). The imported module names, if placed at the top level of a module (outside any functions or classes), are added to the module's global namespace.

What is top level package in Python?

“Top-level code” is the first user-specified Python module that starts running. It's “top-level” because it imports all other modules that the program needs. Sometimes “top-level code” is called an entry point to the application.

Should I use relative or absolute imports Python?

With your new skills, you can confidently import packages and modules from the Python standard library, third party packages, and your own local packages. Remember that you should generally opt for absolute imports over relative ones, unless the path is complex and would make the statement too long.